__   __
  /  \ /  \     ______   _    _     __   __   __
 / /\ / /\ \   /  __  \ | |  | |   /  \  \ \ / /
/ /  / /  \ \  | |  | | | |  | |  / /\ \  \   /
\ \  \ \  / /  | |__| | | |__| | / ____ \  | |
 \ \/ \ \/ /   \_  ___/  \____/ /_/    \_\ |_|
  \__/ \__/      \ \__
                  \___\ by Red Hat
 Build, Store, and Distribute your Containers

Startup timestamp: 
Mon Apr 22 03:32:52 UTC 2024

Running all default registry services without migration
Running init script '/quay-registry/conf/init/certs_install.sh'
Installing extra certificates found in /quay-registry/conf/stack/extra_ca_certs directory
Running init script '/quay-registry/conf/init/client_certs.sh'
Running init script '/quay-registry/conf/init/copy_config_files.sh'
Running init script '/quay-registry/conf/init/d_validate_config_bundle.sh'
Validating Configuration
time="2024-04-22T03:32:55Z" level=debug msg="Validating AccessSettings"
time="2024-04-22T03:32:55Z" level=debug msg="Validating ActionLogArchiving"
time="2024-04-22T03:32:55Z" level=debug msg="Validating AppTokenAuthentication"
time="2024-04-22T03:32:55Z" level=debug msg="Validating BitbucketBuildTrigger"
time="2024-04-22T03:32:55Z" level=debug msg="Validating BuildManager"
time="2024-04-22T03:32:55Z" level=debug msg="Validating Database"
time="2024-04-22T03:32:55Z" level=debug msg="Scheme: postgresql"
time="2024-04-22T03:32:55Z" level=debug msg="Host: quayregistry-quay-database:5432"
time="2024-04-22T03:32:55Z" level=debug msg="Db: quayregistry-quay-database"
time="2024-04-22T03:32:55Z" level=debug msg="Params: "
time="2024-04-22T03:32:55Z" level=debug msg="Including params "
time="2024-04-22T03:32:55Z" level=debug msg="Pinging database at postgresql://quayregistry-quay-database:LR26GYE2Q6C1alDmIITINfDYIVNBINfu7Ep3m38KlTCvZwaP6XwkLo53vn3GRqC-4WmofCbZLtxhHtn8@quayregistry-quay-database:5432/quayregistry-quay-database"
plpgsql
pg_trgm
time="2024-04-22T03:32:55Z" level=debug msg="Validating DistributedStorage"
time="2024-04-22T03:32:55Z" level=debug msg="Validating ElasticSearch"
time="2024-04-22T03:32:55Z" level=debug msg="Validating Email"
time="2024-04-22T03:32:57Z" level=debug msg="Validating GitHubBuildTrigger"
time="2024-04-22T03:32:57Z" level=debug msg="Validating GitHubLogin"
time="2024-04-22T03:32:57Z" level=debug msg="Validating GitLabBuildTrigger"
time="2024-04-22T03:32:57Z" level=debug msg="Validating GoogleLogin"
time="2024-04-22T03:32:57Z" level=debug msg="Validating HostSettings"
time="2024-04-22T03:32:57Z" level=debug msg="Validating JWTAuthentication"
time="2024-04-22T03:32:57Z" level=debug msg="Validating LDAP"
time="2024-04-22T03:32:57Z" level=debug msg="Validating OIDC"
time="2024-04-22T03:32:57Z" level=debug msg="Validating QuayDocumentation"
time="2024-04-22T03:32:57Z" level=debug msg="Validating Redis"
time="2024-04-22T03:32:57Z" level=debug msg="Address: quayregistry-quay-redis:6379"
time="2024-04-22T03:32:57Z" level=debug msg="Username: "
time="2024-04-22T03:32:57Z" level=debug msg="Password Len: 0"
time="2024-04-22T03:32:57Z" level=debug msg="Ssl: <nil>"
time="2024-04-22T03:32:57Z" level=debug msg="Address: quayregistry-quay-redis:6379"
time="2024-04-22T03:32:57Z" level=debug msg="Username: "
time="2024-04-22T03:32:57Z" level=debug msg="Password Len: 0"
time="2024-04-22T03:32:57Z" level=debug msg="Ssl: <nil>"
time="2024-04-22T03:32:57Z" level=debug msg="Validating RepoMirror"
time="2024-04-22T03:32:57Z" level=debug msg="Validating SecurityScanner"
time="2024-04-22T03:32:57Z" level=debug msg="Validating TeamSyncing"
time="2024-04-22T03:32:57Z" level=debug msg="Validating TimeMachine"
time="2024-04-22T03:32:57Z" level=debug msg="Validating UserVisibleSettings"
+------------------------+-------+--------+
|      Field Group       | Error | Status |
+------------------------+-------+--------+
| AccessSettings         | -     | 🟢     |
+------------------------+-------+--------+
| ActionLogArchiving     | -     | 🟢     |
+------------------------+-------+--------+
| AppTokenAuthentication | -     | 🟢     |
+------------------------+-------+--------+
| BitbucketBuildTrigger  | -     | 🟢     |
+------------------------+-------+--------+
| BuildManager           | -     | 🟢     |
+------------------------+-------+--------+
| Database               | -     | 🟢     |
+------------------------+-------+--------+
| DistributedStorage     | -     | 🟢     |
+------------------------+-------+--------+
| ElasticSearch          | -     | 🟢     |
+------------------------+-------+--------+
| Email                  | -     | 🟢     |
+------------------------+-------+--------+
| GitHubBuildTrigger     | -     | 🟢     |
+------------------------+-------+--------+
| GitHubLogin            | -     | 🟢     |
+------------------------+-------+--------+
| GitLabBuildTrigger     | -     | 🟢     |
+------------------------+-------+--------+
| GoogleLogin            | -     | 🟢     |
+------------------------+-------+--------+
| HostSettings           | -     | 🟢     |
+------------------------+-------+--------+
| JWTAuthentication      | -     | 🟢     |
+------------------------+-------+--------+
| LDAP                   | -     | 🟢     |
+------------------------+-------+--------+
| OIDC                   | -     | 🟢     |
+------------------------+-------+--------+
| QuayDocumentation      | -     | 🟢     |
+------------------------+-------+--------+
| Redis                  | -     | 🟢     |
+------------------------+-------+--------+
| RepoMirror             | -     | 🟢     |
+------------------------+-------+--------+
| SecurityScanner        | -     | 🟢     |
+------------------------+-------+--------+
| TeamSyncing            | -     | 🟢     |
+------------------------+-------+--------+
| TimeMachine            | -     | 🟢     |
+------------------------+-------+--------+
| UserVisibleSettings    | -     | 🟢     |
+------------------------+-------+--------+
Running init script '/quay-registry/conf/init/nginx_conf_create.sh'
Running init script '/quay-registry/conf/init/supervisord_conf_create.sh'
Running init script '/quay-registry/conf/init/zz_boot.sh'
/app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if word_sep is not '':
2024-04-22 03:33:00,919 INFO RPC interface 'supervisor' initialized
2024-04-22 03:33:00,919 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2024-04-22 03:33:00,919 INFO supervisord started with pid 7
2024-04-22 03:33:01,921 INFO spawned: 'stdout' with pid 58
2024-04-22 03:33:01,923 INFO spawned: 'autopruneworker' with pid 59
2024-04-22 03:33:01,925 INFO spawned: 'blobuploadcleanupworker' with pid 60
2024-04-22 03:33:01,927 INFO spawned: 'builder' with pid 61
2024-04-22 03:33:01,929 INFO spawned: 'buildlogsarchiver' with pid 62
2024-04-22 03:33:01,931 INFO spawned: 'chunkcleanupworker' with pid 63
2024-04-22 03:33:01,932 INFO spawned: 'dnsmasq' with pid 64
2024-04-22 03:33:01,934 INFO spawned: 'expiredappspecifictokenworker' with pid 65
2024-04-22 03:33:01,937 INFO spawned: 'exportactionlogsworker' with pid 66
2024-04-22 03:33:01,939 INFO spawned: 'gcworker' with pid 67
2024-04-22 03:33:01,941 INFO spawned: 'globalpromstats' with pid 68
2024-04-22 03:33:01,944 INFO spawned: 'gunicorn-registry' with pid 69
2024-04-22 03:33:01,946 INFO spawned: 'gunicorn-secscan' with pid 70
2024-04-22 03:33:02,030 INFO spawned: 'gunicorn-web' with pid 71
2024-04-22 03:33:02,033 INFO spawned: 'logrotateworker' with pid 72
2024-04-22 03:33:02,035 INFO spawned: 'manifestbackfillworker' with pid 73
2024-04-22 03:33:02,038 INFO spawned: 'memcache' with pid 74
2024-04-22 03:33:02,040 INFO spawned: 'namespacegcworker' with pid 75
2024-04-22 03:33:02,044 INFO spawned: 'nginx' with pid 76
2024-04-22 03:33:02,047 INFO spawned: 'notificationworker' with pid 77
2024-04-22 03:33:02,132 INFO spawned: 'pushgateway' with pid 83
2024-04-22 03:33:02,135 INFO spawned: 'queuecleanupworker' with pid 84
2024-04-22 03:33:02,137 INFO spawned: 'quotaregistrysizeworker' with pid 85
2024-04-22 03:33:02,140 INFO spawned: 'quotatotalworker' with pid 86
2024-04-22 03:33:02,230 INFO spawned: 'reconciliationworker' with pid 87
2024-04-22 03:33:02,233 INFO spawned: 'repositoryactioncounter' with pid 90
2024-04-22 03:33:02,236 INFO spawned: 'repositorygcworker' with pid 91
2024-04-22 03:33:02,239 INFO spawned: 'securityscanningnotificationworker' with pid 92
2024-04-22 03:33:02,241 INFO spawned: 'securityworker' with pid 95
2024-04-22 03:33:02,244 INFO spawned: 'servicekey' with pid 96
2024-04-22 03:33:02,246 INFO spawned: 'storagereplication' with pid 97
2024-04-22 03:33:02,332 INFO spawned: 'teamsyncworker' with pid 103
2024-04-22 03:33:03,833 INFO success: stdout entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,833 INFO success: autopruneworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: blobuploadcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: builder entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: buildlogsarchiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: chunkcleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: dnsmasq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: expiredappspecifictokenworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: exportactionlogsworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: gcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: globalpromstats entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: gunicorn-registry entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,834 INFO success: gunicorn-secscan entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: gunicorn-web entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: logrotateworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: manifestbackfillworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: memcache entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: namespacegcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: notificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: pushgateway entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: queuecleanupworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: quotaregistrysizeworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: quotatotalworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,835 INFO success: reconciliationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: repositoryactioncounter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: repositorygcworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: securityscanningnotificationworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: securityworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: servicekey entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: storagereplication entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-04-22 03:33:03,836 INFO success: teamsyncworker entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
dnsmasq stderr | dnsmasq: started, version 2.79 cachesize 150
dnsmasq stderr | dnsmasq: compile time options: IPv6 GNU-getopt DBus no-i18n IDN2 DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth DNSSEC loop-detect inotify
dnsmasq stderr | dnsmasq: reading /etc/resolv.conf
dnsmasq stderr | dnsmasq: using nameserver 172.30.0.10#53
dnsmasq stderr | dnsmasq: read /etc/hosts - 7 addresses
nginx stdout | 2024/04/22 03:33:02 [alert] 106#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 104#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 98#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 108#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 110#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 99#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 114#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 112#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 113#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 101#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 109#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 107#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 111#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 105#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 100#0: setpriority(-10) failed (13: Permission denied)
nginx stdout | 2024/04/22 03:33:02 [alert] 102#0: setpriority(-10) failed (13: Permission denied)
pushgateway stderr | ts=2024-04-22T03:33:02.636Z caller=main.go:86 level=info msg="starting pushgateway" version="(version=, branch=, revision=unknown)"
pushgateway stderr | ts=2024-04-22T03:33:02.636Z caller=main.go:87 level=info build_context="(go=go1.19.13, platform=linux/amd64, user=, date=, tags=unknown)"
pushgateway stderr | ts=2024-04-22T03:33:02.832Z caller=tls_config.go:274 level=info msg="Listening on" address=[::]:9091
pushgateway stderr | ts=2024-04-22T03:33:02.832Z caller=tls_config.go:277 level=info msg="TLS is disabled." http2=false address=[::]:9091
nginx stdout | 2024/04/22 03:33:15 [crit] 106#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.23:8080"
nginx stdout | 2024/04/22 03:33:15 [crit] 106#0: *1 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.23:8080"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:33:15 +0000] "GET /health/instance HTTP/1.1" 502 157 "-" "kube-probe/1.28" (0.000 118 0.000 : 0.000)
chunkcleanupworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
chunkcleanupworker stderr |   if word_sep is not '':
notificationworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
notificationworker stderr |   if word_sep is not '':
buildlogsarchiver stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
buildlogsarchiver stderr |   if word_sep is not '':
securityscanningnotificationworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
securityscanningnotificationworker stderr |   if word_sep is not '':
logrotateworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
logrotateworker stderr |   if word_sep is not '':
nginx stdout | 2024/04/22 03:33:30 [crit] 98#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.23:8080"
nginx stdout | 2024/04/22 03:33:30 [crit] 98#0: *4 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.23:8080"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:33:30 +0000] "GET /health/instance HTTP/1.1" 502 157 "-" "kube-probe/1.28" (0.000 118 0.000 : 0.000)
storagereplication stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
storagereplication stderr |   if word_sep is not '':
teamsyncworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
teamsyncworker stderr |   if word_sep is not '':
autopruneworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
autopruneworker stderr |   if word_sep is not '':
queuecleanupworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
queuecleanupworker stderr |   if word_sep is not '':
quotaregistrysizeworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
quotaregistrysizeworker stderr |   if word_sep is not '':
servicekey stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
servicekey stderr |   if word_sep is not '':
reconciliationworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
reconciliationworker stderr |   if word_sep is not '':
repositorygcworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
repositorygcworker stderr |   if word_sep is not '':
globalpromstats stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
globalpromstats stderr |   if word_sep is not '':
notificationworker stdout | 2024-04-22 03:33:37,233 [77] [DEBUG] [workers.worker] Scheduling worker.
notificationworker stdout | 2024-04-22 03:33:37,235 [77] [INFO] [apscheduler.scheduler] Scheduler started
notificationworker stdout | 2024-04-22 03:33:37,358 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:33:37,358 [77] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
notificationworker stdout | 2024-04-22 03:33:37,430 [77] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:33:42.430973
notificationworker stdout | 2024-04-22 03:33:37,432 [77] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
notificationworker stdout | 2024-04-22 03:33:37,432 [77] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:37:38.433394
notificationworker stdout | 2024-04-22 03:33:37,432 [77] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
notificationworker stdout | 2024-04-22 03:33:37,432 [77] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:33:54.433683
notificationworker stdout | 2024-04-22 03:33:37,432 [77] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
notificationworker stdout | 2024-04-22 03:33:37,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:33:37,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:33:42.430973+00:00 (in 4.997826 seconds)
chunkcleanupworker stdout | 2024-04-22 03:33:38,430 [63] [DEBUG] [__main__] Swift storage not detected; sleeping
expiredappspecifictokenworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
expiredappspecifictokenworker stderr |   if word_sep is not '':
exportactionlogsworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
exportactionlogsworker stderr |   if word_sep is not '':
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,734 [85] [DEBUG] [workers.worker] Scheduling worker.
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,736 [85] [INFO] [apscheduler.scheduler] Scheduler started
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,737 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,737 [85] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:24.738365
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,740 [85] [INFO] [apscheduler.scheduler] Added job "QuotaRegistrySizeWorker._calculate_registry_size" to job store "default"
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,738 [85] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,742 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:33:40,742 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:24.738365+00:00 (in 43.996152 seconds)
gcworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
gcworker stderr |   if word_sep is not '':
logrotateworker stdout | 2024-04-22 03:33:40,934 [72] [DEBUG] [__main__] Action log rotation worker not enabled; skipping
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,741 [92] [DEBUG] [__main__] Starting security scanning notification worker
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,743 [92] [DEBUG] [workers.worker] Scheduling worker.
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,743 [92] [INFO] [apscheduler.scheduler] Scheduler started
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,842 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,842 [92] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,930 [92] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:29.931002
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,932 [92] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,932 [92] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:38:30.933445
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,932 [92] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,932 [92] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:31.933721
securityscanningnotificationworker stdout | 2024-04-22 03:33:41,932 [92] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
securityscanningnotificationworker stdout | 2024-04-22 03:33:42,030 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:33:42,030 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:29.931002+00:00 (in 47.900720 seconds)
buildlogsarchiver stdout | 2024-04-22 03:33:42,030 [62] [DEBUG] [workers.worker] Scheduling worker.
buildlogsarchiver stdout | 2024-04-22 03:33:42,031 [62] [INFO] [apscheduler.scheduler] Scheduler started
buildlogsarchiver stdout | 2024-04-22 03:33:42,044 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:33:42,044 [62] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
buildlogsarchiver stdout | 2024-04-22 03:33:42,130 [62] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:33:58.130973
buildlogsarchiver stdout | 2024-04-22 03:33:42,132 [62] [INFO] [apscheduler.scheduler] Added job "ArchiveBuildLogsWorker._archive_redis_buildlogs" to job store "default"
buildlogsarchiver stdout | 2024-04-22 03:33:42,132 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:33:42,132 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:33:58.130973+00:00 (in 15.998582 seconds)
securityworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
securityworker stderr |   if word_sep is not '':
autopruneworker stdout | 2024-04-22 03:33:42,234 [59] [DEBUG] [workers.worker] Scheduling worker.
autopruneworker stdout | 2024-04-22 03:33:42,236 [59] [INFO] [apscheduler.scheduler] Scheduler started
autopruneworker stdout | 2024-04-22 03:33:42,237 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:33:42,237 [59] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:10.238173
autopruneworker stdout | 2024-04-22 03:33:42,240 [59] [INFO] [apscheduler.scheduler] Added job "AutoPruneWorker.prune" to job store "default"
autopruneworker stdout | 2024-04-22 03:33:42,238 [59] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
autopruneworker stdout | 2024-04-22 03:33:42,241 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:33:42,242 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:10.238173+00:00 (in 27.995942 seconds)
notificationworker stdout | 2024-04-22 03:33:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
storagereplication stdout | 2024-04-22 03:33:42,440 [97] [DEBUG] [__main__] Full storage replication disabled; skipping
notificationworker stdout | 2024-04-22 03:33:42,530 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:33:42 UTC)" (scheduled at 2024-04-22 03:33:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:33:42,530 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:33:52.430973+00:00 (in 9.900459 seconds)
notificationworker stdout | 2024-04-22 03:33:42,531 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:33:42,535 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 33, 42, 533354), True, datetime.datetime(2024, 4, 22, 3, 33, 42, 533354), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:33:42,738 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:33:42,738 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:33:42,738 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:33:52 UTC)" executed successfully
manifestbackfillworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
manifestbackfillworker stderr |   if word_sep is not '':
teamsyncworker stdout | 2024-04-22 03:33:42,836 [103] [DEBUG] [__main__] Team syncing is disabled; sleeping
servicekey stdout | 2024-04-22 03:33:43,237 [96] [DEBUG] [workers.worker] Scheduling worker.
servicekey stdout | 2024-04-22 03:33:43,239 [96] [INFO] [apscheduler.scheduler] Scheduler started
servicekey stdout | 2024-04-22 03:33:43,242 [96] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2024-04-22 03:33:43,242 [96] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 04:26:09.243349
servicekey stdout | 2024-04-22 03:33:43,332 [96] [INFO] [apscheduler.scheduler] Added job "ServiceKeyWorker._refresh_service_key" to job store "default"
servicekey stdout | 2024-04-22 03:33:43,243 [96] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
servicekey stdout | 2024-04-22 03:33:43,333 [96] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2024-04-22 03:33:43,333 [96] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:09.243349+00:00 (in 3145.909422 seconds)
gunicorn-secscan stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
gunicorn-secscan stderr |   if word_sep is not '':
blobuploadcleanupworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
blobuploadcleanupworker stderr |   if word_sep is not '':
repositorygcworker stdout | 2024-04-22 03:33:44,944 [91] [DEBUG] [__main__] Starting repository GC worker
repositorygcworker stdout | 2024-04-22 03:33:45,030 [91] [DEBUG] [workers.worker] Scheduling worker.
repositorygcworker stdout | 2024-04-22 03:33:45,031 [91] [INFO] [apscheduler.scheduler] Scheduler started
repositorygcworker stdout | 2024-04-22 03:33:45,032 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:33:45,032 [91] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:09.033206
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:38:26.035364
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:02.035613
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
repositorygcworker stdout | 2024-04-22 03:33:45,032 [91] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:33:45,034 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:02.035613+00:00 (in 17.000629 seconds)
namespacegcworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
namespacegcworker stderr |   if word_sep is not '':
gunicorn-registry stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
gunicorn-registry stderr |   if word_sep is not '':
nginx stdout | 2024/04/22 03:33:45 [crit] 104#0: *7 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/health/instance", host: "10.129.2.23:8080"
nginx stdout | 2024/04/22 03:33:45 [crit] 104#0: *7 connect() to unix:/tmp/gunicorn_web.sock failed (2: No such file or directory) while connecting to upstream, client: 10.129.2.2, server: , request: "GET /health/instance HTTP/1.1", upstream: "http://unix:/tmp/gunicorn_web.sock:/quay-registry/static/502.html", host: "10.129.2.23:8080"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:33:45 +0000] "GET /health/instance HTTP/1.1" 502 157 "-" "kube-probe/1.28" (0.000 118 0.000 : 0.000)
builder stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
builder stderr |   if word_sep is not '':
gunicorn-web stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
gunicorn-web stderr |   if word_sep is not '':
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,034 [65] [DEBUG] [__main__] Starting expired app specific token GC worker
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,036 [65] [DEBUG] [__main__] Found expiration window: 1d
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,038 [65] [DEBUG] [workers.worker] Scheduling worker.
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,040 [65] [INFO] [apscheduler.scheduler] Scheduler started
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,140 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,141 [65] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:58:12.142129
globalpromstats stdout | 2024-04-22 03:33:47,145 [68] [DEBUG] [workers.worker] Scheduling worker.
globalpromstats stdout | 2024-04-22 03:33:47,145 [68] [INFO] [apscheduler.scheduler] Scheduler started
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,142 [65] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,232 [65] [INFO] [apscheduler.scheduler] Added job "ExpiredAppSpecificTokenWorker._gc_expired_tokens" to job store "default"
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,236 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2024-04-22 03:33:47,237 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:12.142129+00:00 (in 1464.904676 seconds)
repositoryactioncounter stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
repositoryactioncounter stderr |   if word_sep is not '':
globalpromstats stdout | 2024-04-22 03:33:47,331 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2024-04-22 03:33:47,331 [68] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 04:20:15.332514
globalpromstats stdout | 2024-04-22 03:33:47,332 [68] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
quotatotalworker stderr | /app/lib/python3.9/site-packages/netaddr/strategy/__init__.py:189: SyntaxWarning: "is not" with a literal. Did you mean "!="?
quotatotalworker stderr |   if word_sep is not '':
globalpromstats stdout | 2024-04-22 03:33:47,334 [68] [INFO] [apscheduler.scheduler] Added job "GlobalPrometheusStatsWorker._try_report_stats" to job store "default"
globalpromstats stdout | 2024-04-22 03:33:47,337 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2024-04-22 03:33:47,337 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:15.332514+00:00 (in 2787.994574 seconds)
queuecleanupworker stdout | 2024-04-22 03:33:47,443 [84] [DEBUG] [workers.worker] Scheduling worker.
queuecleanupworker stdout | 2024-04-22 03:33:47,445 [84] [INFO] [apscheduler.scheduler] Scheduler started
queuecleanupworker stdout | 2024-04-22 03:33:47,642 [84] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
queuecleanupworker stdout | 2024-04-22 03:33:47,642 [84] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 20:13:20.643404
queuecleanupworker stdout | 2024-04-22 03:33:47,645 [84] [INFO] [apscheduler.scheduler] Added job "QueueCleanupWorker._cleanup_queue" to job store "default"
queuecleanupworker stdout | 2024-04-22 03:33:47,643 [84] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
queuecleanupworker stdout | 2024-04-22 03:33:47,647 [84] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
queuecleanupworker stdout | 2024-04-22 03:33:47,647 [84] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 20:13:20.643404+00:00 (in 59972.996333 seconds)
gcworker stdout | 2024-04-22 03:33:48,836 [67] [DEBUG] [workers.worker] Scheduling worker.
gcworker stdout | 2024-04-22 03:33:48,838 [67] [INFO] [apscheduler.scheduler] Scheduler started
gcworker stdout | 2024-04-22 03:33:48,839 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:33:48,839 [67] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
gcworker stdout | 2024-04-22 03:33:48,841 [67] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:01.842666
gcworker stdout | 2024-04-22 03:33:48,844 [67] [INFO] [apscheduler.scheduler] Added job "GarbageCollectionWorker._garbage_collection_repos" to job store "default"
gcworker stdout | 2024-04-22 03:33:48,845 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:33:48,845 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:01.842666+00:00 (in 12.996954 seconds)
gunicorn-secscan stdout | 2024-04-22 03:33:49,138 [70] [DEBUG] [app] Loading default config.
gunicorn-secscan stdout | 2024-04-22 03:33:49,139 [70] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml
gunicorn-secscan stdout | 2024-04-22 03:33:49,153 [70] [DEBUG] [app] Loaded config
gunicorn-secscan stdout | 2024-04-22 03:33:49,231 [70] [INFO] [util.ipresolver] Loading AWS IP ranges from disk
gunicorn-secscan stdout | 2024-04-22 03:33:49,244 [70] [DEBUG] [util.ipresolver] Building AWS IP ranges
gunicorn-secscan stdout | 2024-04-22 03:33:49,938 [70] [DEBUG] [util.ipresolver] Finished building AWS IP ranges
gunicorn-secscan stdout | 2024-04-22 03:33:50,052 [70] [DEBUG] [data.database] Configuring database
gunicorn-secscan stdout | 2024-04-22 03:33:50,131 [70] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-secscan stdout | 2024-04-22 03:33:50,131 [70] [INFO] [data.secscan_model] ===============================
gunicorn-secscan stdout | 2024-04-22 03:33:50,131 [70] [INFO] [data.secscan_model] Using split secscan model: `[<data.secscan_model.secscan_v4_model.V4SecurityScanner object at 0x7f207404b5b0>]`
gunicorn-secscan stdout | 2024-04-22 03:33:50,132 [70] [INFO] [data.secscan_model] ===============================
gunicorn-secscan stdout | 2024-04-22 03:33:50,132 [70] [DEBUG] [data.logs_model] Configuring log model
gunicorn-secscan stdout | 2024-04-22 03:33:50,132 [70] [INFO] [data.logs_model] ===============================
gunicorn-secscan stdout | 2024-04-22 03:33:50,132 [70] [INFO] [data.logs_model] Using logs model `<data.logs_model.table_logs_model.TableLogsModel object at 0x7f207404b7c0>`
gunicorn-secscan stdout | 2024-04-22 03:33:50,132 [70] [INFO] [data.logs_model] ===============================
manifestbackfillworker stdout | 2024-04-22 03:33:50,235 [73] [DEBUG] [workers.worker] Scheduling worker.
manifestbackfillworker stdout | 2024-04-22 03:33:50,236 [73] [INFO] [apscheduler.scheduler] Scheduler started
manifestbackfillworker stdout | 2024-04-22 03:33:50,236 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2024-04-22 03:33:50,236 [73] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
manifestbackfillworker stdout | 2024-04-22 03:33:50,236 [73] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:47:19.237742
manifestbackfillworker stdout | 2024-04-22 03:33:50,238 [73] [INFO] [apscheduler.scheduler] Added job "ManifestBackfillWorker._backfill_manifests" to job store "default"
manifestbackfillworker stdout | 2024-04-22 03:33:50,238 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2024-04-22 03:33:50,238 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:19.237742+00:00 (in 808.998778 seconds)
gunicorn-registry stdout | 2024-04-22 03:33:50,539 [69] [DEBUG] [app] Loading default config.
gunicorn-registry stdout | 2024-04-22 03:33:50,539 [69] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml
gunicorn-registry stdout | 2024-04-22 03:33:50,634 [69] [DEBUG] [app] Loaded config
gunicorn-registry stdout | 2024-04-22 03:33:50,635 [69] [INFO] [util.ipresolver] Loading AWS IP ranges from disk
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,648 [60] [DEBUG] [workers.worker] Scheduling worker.
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,649 [60] [INFO] [apscheduler.scheduler] Scheduler started
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,649 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,649 [60] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,649 [60] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:58:15.650842
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,730 [60] [INFO] [apscheduler.scheduler] Added job "BlobUploadCleanupWorker._try_cleanup_uploads" to job store "default"
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,730 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2024-04-22 03:33:50,730 [60] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:15.650842+00:00 (in 1464.920422 seconds)
gunicorn-registry stdout | 2024-04-22 03:33:50,731 [69] [DEBUG] [util.ipresolver] Building AWS IP ranges
gunicorn-web stdout | 2024-04-22 03:33:50,933 [71] [DEBUG] [app] Loading default config.
gunicorn-web stdout | 2024-04-22 03:33:50,933 [71] [DEBUG] [util.config.provider.basefileprovider] Applying config file: /quay-registry/conf/stack/config.yaml
gunicorn-web stdout | 2024-04-22 03:33:50,946 [71] [DEBUG] [app] Loaded config
gunicorn-web stdout | 2024-04-22 03:33:50,947 [71] [INFO] [util.ipresolver] Loading AWS IP ranges from disk
gunicorn-secscan stdout | 2024-04-22 03:33:51,040 [70] [DEBUG] [__config__] Starting secscan gunicorn with 2 workers and gevent worker class
gunicorn-web stdout | 2024-04-22 03:33:51,043 [71] [DEBUG] [util.ipresolver] Building AWS IP ranges
gunicorn-registry stdout | 2024-04-22 03:33:51,140 [69] [DEBUG] [util.ipresolver] Finished building AWS IP ranges
namespacegcworker stdout | 2024-04-22 03:33:51,242 [75] [DEBUG] [__main__] Starting namespace GC worker
namespacegcworker stdout | 2024-04-22 03:33:51,243 [75] [DEBUG] [workers.worker] Scheduling worker.
namespacegcworker stdout | 2024-04-22 03:33:51,244 [75] [INFO] [apscheduler.scheduler] Scheduler started
namespacegcworker stdout | 2024-04-22 03:33:51,244 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:33:51,244 [75] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
namespacegcworker stdout | 2024-04-22 03:33:51,244 [75] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:47.245913
namespacegcworker stdout | 2024-04-22 03:33:51,246 [75] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
namespacegcworker stdout | 2024-04-22 03:33:51,246 [75] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:36:30.247852
namespacegcworker stdout | 2024-04-22 03:33:51,247 [75] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
namespacegcworker stdout | 2024-04-22 03:33:51,247 [75] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:10.248668
namespacegcworker stdout | 2024-04-22 03:33:51,247 [75] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
namespacegcworker stdout | 2024-04-22 03:33:51,247 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:33:51,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:10.248668+00:00 (in 19.000573 seconds)
namespacegcworker stdout | 2024-04-22 03:33:51,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:33:51,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:10.248668+00:00 (in 19.000448 seconds)
builder stdout | 2024-04-22 03:33:51,437 [61] [DEBUG] [__main__] Building is disabled. Please enable the feature flag
gunicorn-web stdout | 2024-04-22 03:33:51,444 [71] [DEBUG] [util.ipresolver] Finished building AWS IP ranges
gunicorn-registry stdout | 2024-04-22 03:33:51,454 [69] [DEBUG] [data.database] Configuring database
gunicorn-registry stdout | 2024-04-22 03:33:51,454 [69] [INFO] [data.database] Connection pooling enabled for postgresql; stale timeout: None; max connection count: None
gunicorn-registry stdout | 2024-04-22 03:33:51,455 [69] [INFO] [data.secscan_model] ===============================
gunicorn-registry stdout | 2024-04-22 03:33:51,455 [69] [INFO] [data.secscan_model] Using split secscan model: `[<data.secscan_model.secscan_v4_model.V4SecurityScanner object at 0x7f748adf8a60>]`
gunicorn-registry stdout | 2024-04-22 03:33:51,455 [69] [INFO] [data.secscan_model] ===============================
gunicorn-registry stdout | 2024-04-22 03:33:51,455 [69] [DEBUG] [data.logs_model] Configuring log model
gunicorn-registry stdout | 2024-04-22 03:33:51,455 [69] [INFO] [data.logs_model] ===============================
gunicorn-registry stdout | 2024-04-22 03:33:51,455 [69] [INFO] [data.logs_model] Using logs model `<data.logs_model.table_logs_model.TableLogsModel object at 0x7f748adf8d90>`
gunicorn-registry stdout | 2024-04-22 03:33:51,456 [69] [INFO] [data.logs_model] ===============================
gunicorn-web stdout | 2024-04-22 03:33:51,461 [71] [DEBUG] [data.database] Configuring database
gunicorn-web stdout | 2024-04-22 03:33:51,530 [71] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:33:51,530 [71] [INFO] [data.secscan_model] ===============================
gunicorn-web stdout | 2024-04-22 03:33:51,530 [71] [INFO] [data.secscan_model] Using split secscan model: `[<data.secscan_model.secscan_v4_model.V4SecurityScanner object at 0x7fabee7b7760>]`
gunicorn-web stdout | 2024-04-22 03:33:51,531 [71] [INFO] [data.secscan_model] ===============================
gunicorn-web stdout | 2024-04-22 03:33:51,531 [71] [DEBUG] [data.logs_model] Configuring log model
gunicorn-web stdout | 2024-04-22 03:33:51,531 [71] [INFO] [data.logs_model] ===============================
gunicorn-web stdout | 2024-04-22 03:33:51,531 [71] [INFO] [data.logs_model] Using logs model `<data.logs_model.table_logs_model.TableLogsModel object at 0x7fabee7b79a0>`
gunicorn-web stdout | 2024-04-22 03:33:51,531 [71] [INFO] [data.logs_model] ===============================
repositoryactioncounter stdout | 2024-04-22 03:33:51,558 [90] [DEBUG] [workers.worker] Scheduling worker.
repositoryactioncounter stdout | 2024-04-22 03:33:51,558 [90] [INFO] [apscheduler.scheduler] Scheduler started
repositoryactioncounter stdout | 2024-04-22 03:33:51,559 [90] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositoryactioncounter stdout | 2024-04-22 03:33:51,559 [90] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
repositoryactioncounter stdout | 2024-04-22 03:33:51,559 [90] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 07:30:50.560310
repositoryactioncounter stdout | 2024-04-22 03:33:51,630 [90] [INFO] [apscheduler.scheduler] Added job "RepositoryActionCountWorker._run_counting" to job store "default"
repositoryactioncounter stdout | 2024-04-22 03:33:51,630 [90] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositoryactioncounter stdout | 2024-04-22 03:33:51,630 [90] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 07:30:50.560310+00:00 (in 14218.929385 seconds)
exportactionlogsworker stdout | 2024-04-22 03:33:52,042 [66] [DEBUG] [__main__] Starting export action logs worker
exportactionlogsworker stdout | 2024-04-22 03:33:52,043 [66] [DEBUG] [workers.worker] Scheduling worker.
exportactionlogsworker stdout | 2024-04-22 03:33:52,044 [66] [INFO] [apscheduler.scheduler] Scheduler started
exportactionlogsworker stdout | 2024-04-22 03:33:52,044 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:33:52,044 [66] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
exportactionlogsworker stdout | 2024-04-22 03:33:52,044 [66] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:33:53.045554
exportactionlogsworker stdout | 2024-04-22 03:33:52,045 [66] [INFO] [apscheduler.scheduler] Added job "QueueWorker.poll_queue" to job store "default"
exportactionlogsworker stdout | 2024-04-22 03:33:52,045 [66] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:37:23.046900
exportactionlogsworker stdout | 2024-04-22 03:33:52,046 [66] [INFO] [apscheduler.scheduler] Added job "QueueWorker.update_queue_metrics" to job store "default"
exportactionlogsworker stdout | 2024-04-22 03:33:52,046 [66] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:18.047111
exportactionlogsworker stdout | 2024-04-22 03:33:52,046 [66] [INFO] [apscheduler.scheduler] Added job "QueueWorker.run_watchdog" to job store "default"
exportactionlogsworker stdout | 2024-04-22 03:33:52,046 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:33:52,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:33:53.045554+00:00 (in 0.999158 seconds)
securityworker stdout | 2024-04-22 03:33:52,063 [95] [DEBUG] [workers.worker] Scheduling worker.
securityworker stdout | 2024-04-22 03:33:52,064 [95] [INFO] [apscheduler.scheduler] Scheduler started
securityworker stdout | 2024-04-22 03:33:52,064 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:33:52,064 [95] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
securityworker stdout | 2024-04-22 03:33:52,064 [95] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:17.065745
securityworker stdout | 2024-04-22 03:33:52,065 [95] [INFO] [apscheduler.scheduler] Added job "SecurityWorker._index_in_scanner" to job store "default"
securityworker stdout | 2024-04-22 03:33:52,066 [95] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 03:34:14.067050
securityworker stdout | 2024-04-22 03:33:52,066 [95] [INFO] [apscheduler.scheduler] Added job "SecurityWorker._index_recent_manifests_in_scanner" to job store "default"
securityworker stdout | 2024-04-22 03:33:52,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:33:52,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:14.067050+00:00 (in 22.000692 seconds)
notificationworker stdout | 2024-04-22 03:33:52,432 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:33:52,432 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:33:54.433683+00:00 (in 2.001032 seconds)
notificationworker stdout | 2024-04-22 03:33:52,432 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:02 UTC)" (scheduled at 2024-04-22 03:33:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:33:52,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:33:52,434 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 33, 52, 432969), True, datetime.datetime(2024, 4, 22, 3, 33, 52, 432969), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:33:52,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:33:52,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:33:52,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:02 UTC)" executed successfully
gunicorn-registry stdout | 2024-04-22 03:33:52,570 [69] [DEBUG] [__config__] Starting registry gunicorn with 8 workers and gevent worker class
gunicorn-web stdout | 2024-04-22 03:33:52,804 [71] [DEBUG] [__config__] Starting web gunicorn with 4 workers and gevent worker class
exportactionlogsworker stdout | 2024-04-22 03:33:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:33:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:33:53 UTC)" (scheduled at 2024-04-22 03:33:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:33:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:33:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 33, 53, 46339), True, datetime.datetime(2024, 4, 22, 3, 33, 53, 46339), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:33:53,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:18.047111+00:00 (in 24.999298 seconds)
exportactionlogsworker stdout | 2024-04-22 03:33:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:33:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:33:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:33:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:33:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:02.430973+00:00 (in 7.996713 seconds)
notificationworker stdout | 2024-04-22 03:33:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:54 UTC)" (scheduled at 2024-04-22 03:33:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:33:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:33:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:33:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:33:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:33:58 UTC)" (scheduled at 2024-04-22 03:33:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:33:58,133 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 33, 58, 132124), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:33:58,133 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:28.130973+00:00 (in 29.997246 seconds)
buildlogsarchiver stdout | 2024-04-22 03:33:58,151 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:33:58,152 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:33:58,152 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:34:00,636 [282] [DEBUG] [app] Starting request: urn:request:5a2c0fb8-e14e-454c-bd29-bc0acd7d5574 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:00,645 [282] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-registry stdout | 2024-04-22 03:34:00,673 [275] [DEBUG] [app] Starting request: urn:request:fa750492-9358-40cd-8911-868394b57864 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:00,676 [275] [DEBUG] [app] Ending request: urn:request:fa750492-9358-40cd-8911-868394b57864 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fa750492-9358-40cd-8911-868394b57864', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.009 162 0.008)
gunicorn-web stdout | 2024-04-22 03:34:00,678 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:34:00,680 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:00,681 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:00,688 [284] [DEBUG] [app] Starting request: urn:request:578d55c6-d991-41e2-befa-8605d09fbb80 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:00,690 [284] [DEBUG] [app] Ending request: urn:request:578d55c6-d991-41e2-befa-8605d09fbb80 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:578d55c6-d991-41e2-befa-8605d09fbb80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.009 159 0.009)
gunicorn-web stdout | 2024-04-22 03:34:00,692 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:00,692 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:00,696 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2024, 4, 22, 3, 34, 0, 694205), None, 'quay', 'quay', datetime.datetime(2024, 4, 22, 3, 34, 0, 694246), 'quay', None, datetime.datetime(2024, 4, 21, 3, 34, 0, 694266), datetime.datetime(2024, 4, 15, 3, 34, 0, 694279), None])
gunicorn-web stdout | 2024-04-22 03:34:00,714 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:00,724 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:00,730 [282] [DEBUG] [app] Ending request: urn:request:5a2c0fb8-e14e-454c-bd29-bc0acd7d5574 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5a2c0fb8-e14e-454c-bd29-bc0acd7d5574', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:00,731 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:34:00,732 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.100 118 0.100)
gunicorn-web stdout | 2024-04-22 03:34:00,734 [282] [DEBUG] [app] Starting request: urn:request:d84c02cf-5726-4cf7-8603-1adf62f946fe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-registry stdout | 2024-04-22 03:34:00,741 [276] [DEBUG] [app] Starting request: urn:request:4b0b7c2d-89bd-492b-96b9-17c5370f539e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:00,742 [276] [DEBUG] [app] Ending request: urn:request:4b0b7c2d-89bd-492b-96b9-17c5370f539e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4b0b7c2d-89bd-492b-96b9-17c5370f539e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.006 162 0.006)
gunicorn-registry stdout | 2024-04-22 03:34:00,743 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:00,744 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:00,746 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:00,748 [282] [DEBUG] [app] Starting request: urn:request:53fa6905-1ccb-4d8f-96d3-5d326dc55b8b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:00,749 [282] [DEBUG] [app] Ending request: urn:request:53fa6905-1ccb-4d8f-96d3-5d326dc55b8b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:53fa6905-1ccb-4d8f-96d3-5d326dc55b8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:34:00,749 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:00,749 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:00,750 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:00,759 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:00,771 [282] [DEBUG] [app] Ending request: urn:request:d84c02cf-5726-4cf7-8603-1adf62f946fe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d84c02cf-5726-4cf7-8603-1adf62f946fe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:00,771 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:34:00,772 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.039)
gcworker stdout | 2024-04-22 03:34:01,843 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:34:01,844 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:01 UTC)" (scheduled at 2024-04-22 03:34:01.842666+00:00)
gcworker stdout | 2024-04-22 03:34:01,844 [67] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2024-04-22 03:34:01,845 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:31.842666+00:00 (in 29.997672 seconds)
gcworker stdout | 2024-04-22 03:34:01,858 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:34:01,858 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:34:01,858 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:34:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:34:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:02 UTC)" (scheduled at 2024-04-22 03:34:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:34:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:34:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:02 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:34:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:09.033206+00:00 (in 6.996488 seconds)
notificationworker stdout | 2024-04-22 03:34:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:12.430973+00:00 (in 9.999493 seconds)
notificationworker stdout | 2024-04-22 03:34:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:12 UTC)" (scheduled at 2024-04-22 03:34:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:34:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:34:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 2, 431766), True, datetime.datetime(2024, 4, 22, 3, 34, 2, 431766), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:34:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:34:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:34:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:34:03,178 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:34:04,379 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:34:07,064 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:34:08,268 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:34:08,647 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:34:08,665 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:34:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:34:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:02.035613+00:00 (in 53.001956 seconds)
repositorygcworker stdout | 2024-04-22 03:34:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:09 UTC)" (scheduled at 2024-04-22 03:34:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:34:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:34:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 9, 33929), True, datetime.datetime(2024, 4, 22, 3, 34, 9, 33929), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:34:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:34:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:34:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:34:09,272 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:34:09,948 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:34:09,966 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
autopruneworker stdout | 2024-04-22 03:34:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:34:10,239 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:10 UTC)" (scheduled at 2024-04-22 03:34:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:34:10,239 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:40.238173+00:00 (in 29.998481 seconds)
namespacegcworker stdout | 2024-04-22 03:34:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:34:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:10 UTC)" (scheduled at 2024-04-22 03:34:10.248668+00:00)
autopruneworker stdout | 2024-04-22 03:34:10,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756790248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:34:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:47.245913+00:00 (in 36.996267 seconds)
namespacegcworker stdout | 2024-04-22 03:34:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:34:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:34:10,256 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:34:10,256 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:34:10,256 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:40 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:34:11,364 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:34:12,234 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:34:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:22.430973+00:00 (in 9.999487 seconds)
notificationworker stdout | 2024-04-22 03:34:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:22 UTC)" (scheduled at 2024-04-22 03:34:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:34:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:34:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 12, 431749), True, datetime.datetime(2024, 4, 22, 3, 34, 12, 431749), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:34:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:34:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:34:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:22 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:34:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:34:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:14 UTC)" (scheduled at 2024-04-22 03:34:14.067050+00:00)
securityworker stdout | 2024-04-22 03:34:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:34:14,069 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:17.065745+00:00 (in 2.996265 seconds)
securityworker stdout | 2024-04-22 03:34:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:34:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:34:14,073 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:14,073 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:34:14,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:34:14,160 [95] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): quayregistry-clair-app.quay-enterprise.svc.cluster.local:80
securityworker stdout | 2024-04-22 03:34:14,172 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:34:14,175 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:34:14,191 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:34:14,197 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:34:14,197 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:34:14,198 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 29, 14, 174655), 1, 2])
securityworker stdout | 2024-04-22 03:34:14,202 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:34:14,202 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:34:14,204 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 29, 14, 174655), 1, 2])
securityworker stdout | 2024-04-22 03:34:14,208 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:34:14,208 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:34:14,208 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:14,208 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:34:14,209 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:34:14,209 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:34:14,209 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:34:14,213 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:14,213 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:34:14,213 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:34:14,679 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:34:14,967 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:34:15,359 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:34:15,636 [286] [DEBUG] [app] Starting request: urn:request:9ed7ccb9-f5df-41b1-9b42-fd6144def443 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:15,636 [285] [DEBUG] [app] Starting request: urn:request:c87f6536-c237-411a-8cc1-1e772670dd42 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:15,644 [286] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-web stdout | 2024-04-22 03:34:15,644 [285] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-registry stdout | 2024-04-22 03:34:15,662 [275] [DEBUG] [app] Starting request: urn:request:a05b4218-186e-448d-9cab-4ae1a6165d19 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:15,662 [275] [DEBUG] [app] Ending request: urn:request:a05b4218-186e-448d-9cab-4ae1a6165d19 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a05b4218-186e-448d-9cab-4ae1a6165d19', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:34:15,663 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:34:15,663 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:34:15,664 [281] [DEBUG] [app] Starting request: urn:request:48834680-55ff-4207-ada9-56035eb97cc8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:15,665 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:34:15,665 [281] [DEBUG] [app] Ending request: urn:request:48834680-55ff-4207-ada9-56035eb97cc8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48834680-55ff-4207-ada9-56035eb97cc8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.005 162 0.005)
gunicorn-registry stdout | 2024-04-22 03:34:15,666 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:15,666 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:15,666 [286] [DEBUG] [app] Starting request: urn:request:1c5385ec-77f9-490e-8d12-bfc370e03398 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:15,667 [286] [DEBUG] [app] Ending request: urn:request:1c5385ec-77f9-490e-8d12-bfc370e03398 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1c5385ec-77f9-490e-8d12-bfc370e03398', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:34:15,668 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.002)
gunicorn-web stdout | 2024-04-22 03:34:15,668 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:15,668 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:15,669 [282] [DEBUG] [app] Starting request: urn:request:6cb699d4-4431-49ae-9b1a-dd9ec6403cd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:15,669 [282] [DEBUG] [app] Ending request: urn:request:6cb699d4-4431-49ae-9b1a-dd9ec6403cd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6cb699d4-4431-49ae-9b1a-dd9ec6403cd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:34:15,670 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:15,670 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:15,671 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2024, 4, 22, 3, 34, 15, 669975), None, 'quay', 'quay', datetime.datetime(2024, 4, 22, 3, 34, 15, 670004), 'quay', None, datetime.datetime(2024, 4, 21, 3, 34, 15, 670019), datetime.datetime(2024, 4, 15, 3, 34, 15, 670028), None])
gunicorn-web stdout | 2024-04-22 03:34:15,672 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2024, 4, 22, 3, 34, 15, 671267), None, 'quay', 'quay', datetime.datetime(2024, 4, 22, 3, 34, 15, 671290), 'quay', None, datetime.datetime(2024, 4, 21, 3, 34, 15, 671303), datetime.datetime(2024, 4, 15, 3, 34, 15, 671312), None])
gunicorn-web stdout | 2024-04-22 03:34:15,686 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:15,686 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:15,695 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:15,695 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:15,699 [286] [DEBUG] [app] Ending request: urn:request:9ed7ccb9-f5df-41b1-9b42-fd6144def443 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9ed7ccb9-f5df-41b1-9b42-fd6144def443', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:15,699 [285] [DEBUG] [app] Ending request: urn:request:c87f6536-c237-411a-8cc1-1e772670dd42 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c87f6536-c237-411a-8cc1-1e772670dd42', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:15,699 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:34:15,699 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:34:15,699 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.068 118 0.068)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.069 118 0.069)
gunicorn-web stdout | 2024-04-22 03:34:15,700 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:34:16,076 [282] [DEBUG] [app] Starting request: urn:request:2e590db8-ab5c-4026-aaaf-b3417f27927a (/) {'X-Forwarded-For': '10.128.2.20'}
gunicorn-web stdout | 2024-04-22 03:34:16,085 [282] [DEBUG] [endpoints.api.discovery] Unable to find method for POST in class <class 'endpoints.api.repository.RepositoryTrust'>
gunicorn-web stdout | 2024-04-22 03:34:16,111 [282] [DEBUG] [app] Ending request: urn:request:2e590db8-ab5c-4026-aaaf-b3417f27927a (/) {'endpoint': 'web.index', 'request_id': 'urn:request:2e590db8-ab5c-4026-aaaf-b3417f27927a', 'remote_addr': '10.128.2.20', 'http_method': 'GET', 'original_url': 'http://quayregistry-quay-app/', 'path': '/', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'curl/7.61.1'}
gunicorn-web stdout | 2024-04-22 03:34:16,112 [282] [INFO] [gunicorn.access] 10.128.2.20 - - [22/Apr/2024:03:34:16 +0000] "GET / HTTP/1.0" 200 93195 "-" "curl/7.61.1"
nginx stdout | 10.128.2.20 (-) - - [22/Apr/2024:03:34:16 +0000] "GET / HTTP/1.1" 200 93195 "-" "curl/7.61.1" (0.040 85 0.037)
exportactionlogsworker stdout | 2024-04-22 03:34:16,167 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:34:17,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:34:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:44.067050+00:00 (in 27.000575 seconds)
securityworker stdout | 2024-04-22 03:34:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:47 UTC)" (scheduled at 2024-04-22 03:34:17.065745+00:00)
securityworker stdout | 2024-04-22 03:34:17,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:34:17,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:34:17,070 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:34:17,072 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:34:17,085 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:34:17,089 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:34:17,089 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:34:17,266 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:34:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:34:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:53.045554+00:00 (in 34.997977 seconds)
exportactionlogsworker stdout | 2024-04-22 03:34:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:18 UTC)" (scheduled at 2024-04-22 03:34:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:34:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:34:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:34:18,070 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:34:18,566 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:34:19,252 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:34:19,253 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:34:19,255 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:34:19,679 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:34:20,172 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:34:20,248 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:34:20,666 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:34:20,735 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:34:20,740 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:34:20,743 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:34:20,744 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:34:20,746 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:34:20,748 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:34:20,751 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:34:20,753 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:34:20,959 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:34:20,967 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:34:20,972 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
repositoryactioncounter stdout | 2024-04-22 03:34:20,980 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
gunicorn-web stdout | 2024-04-22 03:34:20,984 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:34:20,987 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 03:34:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:32.430973+00:00 (in 9.999493 seconds)
notificationworker stdout | 2024-04-22 03:34:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:32 UTC)" (scheduled at 2024-04-22 03:34:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:34:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:34:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 22, 431790), True, datetime.datetime(2024, 4, 22, 3, 34, 22, 431790), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:34:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:34:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:34:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:34:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:34:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:24 UTC)" (scheduled at 2024-04-22 03:34:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:34:24,740 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:34:24,740 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:24.738365+00:00 (in 59.997973 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:34:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:34:24,753 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:24 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:34:25,972 [284] [DEBUG] [app] Starting request: urn:request:45d54d86-ba99-48ba-952f-984b7da39fa0 (/) {'X-Forwarded-For': '10.129.2.24'}
gunicorn-web stdout | 2024-04-22 03:34:25,981 [284] [DEBUG] [endpoints.api.discovery] Unable to find method for POST in class <class 'endpoints.api.repository.RepositoryTrust'>
gunicorn-web stdout | 2024-04-22 03:34:26,008 [284] [DEBUG] [app] Ending request: urn:request:45d54d86-ba99-48ba-952f-984b7da39fa0 (/) {'endpoint': 'web.index', 'request_id': 'urn:request:45d54d86-ba99-48ba-952f-984b7da39fa0', 'remote_addr': '10.129.2.24', 'http_method': 'GET', 'original_url': 'http://quayregistry-quay-app/', 'path': '/', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'curl/7.61.1'}
gunicorn-web stdout | 2024-04-22 03:34:26,009 [284] [INFO] [gunicorn.access] 10.129.2.24 - - [22/Apr/2024:03:34:26 +0000] "GET / HTTP/1.0" 200 93195 "-" "curl/7.61.1"
nginx stdout | 10.129.2.24 (-) - - [22/Apr/2024:03:34:26 +0000] "GET / HTTP/1.1" 200 93195 "-" "curl/7.61.1" (0.038 85 0.038)
buildlogsarchiver stdout | 2024-04-22 03:34:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:34:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:58.130973+00:00 (in 29.998991 seconds)
buildlogsarchiver stdout | 2024-04-22 03:34:28,132 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:58 UTC)" (scheduled at 2024-04-22 03:34:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:34:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 34, 28, 132242), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:34:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:34:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:34:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:34:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,932 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:34:29 UTC)" (scheduled at 2024-04-22 03:34:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,932 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,934 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 29, 932627), True, datetime.datetime(2024, 4, 22, 3, 34, 29, 932627), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:31.933721+00:00 (in 1.998943 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,949 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,949 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:34:29,950 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:34:30,632 [282] [DEBUG] [app] Starting request: urn:request:a785956d-5cf2-4c25-a1f4-d2bfc6519585 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:30,633 [285] [DEBUG] [app] Starting request: urn:request:e27ef327-d9ef-4ad0-a012-e392f3942092 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:30,635 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:34:30,637 [275] [DEBUG] [app] Starting request: urn:request:cc64ce1a-4a90-4704-ae1c-aa284572e5d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:30,637 [281] [DEBUG] [app] Starting request: urn:request:81c5ef8c-743c-4544-89fa-9b841d537615 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:30,637 [275] [DEBUG] [app] Ending request: urn:request:cc64ce1a-4a90-4704-ae1c-aa284572e5d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc64ce1a-4a90-4704-ae1c-aa284572e5d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:34:30,637 [281] [DEBUG] [app] Ending request: urn:request:81c5ef8c-743c-4544-89fa-9b841d537615 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81c5ef8c-743c-4544-89fa-9b841d537615', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:34:30,638 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:30,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:34:30,638 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:34:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:30,640 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:30,641 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:30,642 [282] [DEBUG] [app] Starting request: urn:request:3bd301c5-c362-4c42-8664-cebf9ed77969 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:30,643 [285] [DEBUG] [app] Starting request: urn:request:6e157f7b-35df-4626-afe4-0b858664a245 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:30,643 [282] [DEBUG] [app] Ending request: urn:request:3bd301c5-c362-4c42-8664-cebf9ed77969 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3bd301c5-c362-4c42-8664-cebf9ed77969', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:34:30,643 [285] [DEBUG] [app] Ending request: urn:request:6e157f7b-35df-4626-afe4-0b858664a245 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6e157f7b-35df-4626-afe4-0b858664a245', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:34:30,643 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:34:30,643 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:30,644 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:34:30,644 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:30,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:30,645 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:30,654 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:30,654 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:30,666 [282] [DEBUG] [app] Ending request: urn:request:a785956d-5cf2-4c25-a1f4-d2bfc6519585 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a785956d-5cf2-4c25-a1f4-d2bfc6519585', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:30,667 [285] [DEBUG] [app] Ending request: urn:request:e27ef327-d9ef-4ad0-a012-e392f3942092 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e27ef327-d9ef-4ad0-a012-e392f3942092', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:30,667 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:34:30,667 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:34:30,667 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:34:30,667 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.036)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.035)
gcworker stdout | 2024-04-22 03:34:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:34:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:01.842666+00:00 (in 29.999458 seconds)
gcworker stdout | 2024-04-22 03:34:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:01 UTC)" (scheduled at 2024-04-22 03:34:31.842666+00:00)
gcworker stdout | 2024-04-22 03:34:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:34:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:34:31,934 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:34:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:29.931002+00:00 (in 57.996485 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:34:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:31 UTC)" (scheduled at 2024-04-22 03:34:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:34:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:34:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:34:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:42.430973+00:00 (in 9.999366 seconds)
notificationworker stdout | 2024-04-22 03:34:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:42 UTC)" (scheduled at 2024-04-22 03:34:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:34:32,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:34:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 32, 432063), True, datetime.datetime(2024, 4, 22, 3, 34, 32, 432063), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:34:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:34:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:34:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:34:33,214 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:34:34,418 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:34:37,096 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:34:38,305 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:34:38,659 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:34:38,672 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:34:39,309 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:34:39,983 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:34:40,003 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
autopruneworker stdout | 2024-04-22 03:34:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:34:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:10.238173+00:00 (in 29.999391 seconds)
autopruneworker stdout | 2024-04-22 03:34:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:10 UTC)" (scheduled at 2024-04-22 03:34:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:34:40,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756820248, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:34:40,256 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:34:40,256 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:34:40,256 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:10 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:34:41,404 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:34:42,269 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:34:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:52.430973+00:00 (in 9.999420 seconds)
notificationworker stdout | 2024-04-22 03:34:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:52 UTC)" (scheduled at 2024-04-22 03:34:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:34:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:34:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 42, 431981), True, datetime.datetime(2024, 4, 22, 3, 34, 42, 431981), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:34:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:34:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:34:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:34:52 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:34:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:34:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:47.065745+00:00 (in 2.998104 seconds)
securityworker stdout | 2024-04-22 03:34:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:14 UTC)" (scheduled at 2024-04-22 03:34:44.067050+00:00)
securityworker stdout | 2024-04-22 03:34:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:34:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:34:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:34:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:34:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:34:44,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:34:44,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:34:44,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:34:44,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:34:44,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:34:44,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 29, 44, 76689), 1, 2])
securityworker stdout | 2024-04-22 03:34:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:34:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:34:44,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 29, 44, 76689), 1, 2])
securityworker stdout | 2024-04-22 03:34:44,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:34:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:34:44,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:44,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:34:44,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:34:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:34:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:34:44,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:34:44,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:34:44,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:34:44,709 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:34:45,003 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:34:45,396 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:34:45,632 [286] [DEBUG] [app] Starting request: urn:request:6f56a945-0037-4c9d-b3d9-9413e35d0915 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:45,632 [282] [DEBUG] [app] Starting request: urn:request:2e51c3ee-d01a-4075-a2d2-d7490f1eb0a8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:34:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:34:45,636 [275] [DEBUG] [app] Starting request: urn:request:bc30c061-420b-4b9c-9965-5d8b5909bdf0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:45,636 [281] [DEBUG] [app] Starting request: urn:request:6537ba94-cfb6-4f1f-8536-bad5777600f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:34:45,636 [275] [DEBUG] [app] Ending request: urn:request:bc30c061-420b-4b9c-9965-5d8b5909bdf0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bc30c061-420b-4b9c-9965-5d8b5909bdf0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:34:45,636 [281] [DEBUG] [app] Ending request: urn:request:6537ba94-cfb6-4f1f-8536-bad5777600f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6537ba94-cfb6-4f1f-8536-bad5777600f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:34:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:34:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:34:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:34:45,639 [286] [DEBUG] [app] Starting request: urn:request:7c0c3f66-a100-4aa7-b307-093a36ba4b1b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:45,639 [286] [DEBUG] [app] Ending request: urn:request:7c0c3f66-a100-4aa7-b307-093a36ba4b1b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7c0c3f66-a100-4aa7-b307-093a36ba4b1b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:34:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:34:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:45,640 [284] [DEBUG] [app] Starting request: urn:request:a9c702e3-1a1a-4864-b7b2-296669e93dac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:34:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:45,640 [284] [DEBUG] [app] Ending request: urn:request:a9c702e3-1a1a-4864-b7b2-296669e93dac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a9c702e3-1a1a-4864-b7b2-296669e93dac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:34:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:34:45,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:34:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:34:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:34:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:34:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:34:45,661 [286] [DEBUG] [app] Ending request: urn:request:6f56a945-0037-4c9d-b3d9-9413e35d0915 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6f56a945-0037-4c9d-b3d9-9413e35d0915', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:34:45,661 [282] [DEBUG] [app] Ending request: urn:request:2e51c3ee-d01a-4075-a2d2-d7490f1eb0a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2e51c3ee-d01a-4075-a2d2-d7490f1eb0a8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:34:45,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:34:45,662 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:34:45,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:34:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:34:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
exportactionlogsworker stdout | 2024-04-22 03:34:46,204 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:34:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:34:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:14.067050+00:00 (in 27.000814 seconds)
securityworker stdout | 2024-04-22 03:34:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:17 UTC)" (scheduled at 2024-04-22 03:34:47.065745+00:00)
securityworker stdout | 2024-04-22 03:34:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:34:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:34:47,068 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:34:47,069 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:34:47,081 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:34:47,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:34:47,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:34:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:34:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:10.248668+00:00 (in 23.002145 seconds)
namespacegcworker stdout | 2024-04-22 03:34:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:47 UTC)" (scheduled at 2024-04-22 03:34:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:34:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:34:47,248 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 47, 246920), True, datetime.datetime(2024, 4, 22, 3, 34, 47, 246920), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:34:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:34:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:34:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:34:47,303 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:34:48,102 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:34:48,602 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:34:49,263 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:34:49,269 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:34:49,273 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:34:49,715 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:34:50,189 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:34:50,283 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:34:50,677 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:34:50,747 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:34:50,753 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:34:50,755 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:34:50,758 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:34:50,760 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:34:50,764 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:34:50,767 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:34:50,770 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:34:50,969 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:34:50,975 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:34:50,979 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:34:50,991 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:34:50,995 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:34:51,017 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:34:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:34:54.433683+00:00 (in 2.002095 seconds)
notificationworker stdout | 2024-04-22 03:34:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:02 UTC)" (scheduled at 2024-04-22 03:34:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:34:52,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:34:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 52, 432196), True, datetime.datetime(2024, 4, 22, 3, 34, 52, 432196), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:34:52,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:34:52,450 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:34:52,450 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:34:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:34:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:18.047111+00:00 (in 25.001058 seconds)
exportactionlogsworker stdout | 2024-04-22 03:34:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:53 UTC)" (scheduled at 2024-04-22 03:34:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:34:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:34:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 34, 53, 46385), True, datetime.datetime(2024, 4, 22, 3, 34, 53, 46385), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:34:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:34:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:34:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:34:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:34:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:02.430973+00:00 (in 7.996636 seconds)
notificationworker stdout | 2024-04-22 03:34:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:54 UTC)" (scheduled at 2024-04-22 03:34:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:34:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:34:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:35:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:34:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:34:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:28.130973+00:00 (in 29.999461 seconds)
buildlogsarchiver stdout | 2024-04-22 03:34:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:28 UTC)" (scheduled at 2024-04-22 03:34:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:34:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 34, 58, 131732), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:34:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:34:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:34:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:35:00,632 [284] [DEBUG] [app] Starting request: urn:request:42343c8f-fd0e-4b0f-b4b4-904943560034 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:00,632 [286] [DEBUG] [app] Starting request: urn:request:e0a7aae8-49a9-4fe3-96fc-47e567c744fd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:00,636 [275] [DEBUG] [app] Starting request: urn:request:3ad82b16-1259-4496-a723-5c85b5ffa65d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:35:00,636 [275] [DEBUG] [app] Ending request: urn:request:3ad82b16-1259-4496-a723-5c85b5ffa65d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ad82b16-1259-4496-a723-5c85b5ffa65d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:35:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:00,637 [284] [DEBUG] [urllib3.connectionpool] Starting new HTTP connection (1): localhost:8080
gunicorn-web stdout | 2024-04-22 03:35:00,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:00,639 [281] [DEBUG] [app] Starting request: urn:request:7161a350-218b-4a05-8ba9-b09d017a4da7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:35:00,639 [281] [DEBUG] [app] Ending request: urn:request:7161a350-218b-4a05-8ba9-b09d017a4da7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7161a350-218b-4a05-8ba9-b09d017a4da7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:35:00,640 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:35:00,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:00,640 [282] [DEBUG] [app] Starting request: urn:request:b5626a2b-7210-41c6-ba3f-e54d5c62e478 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:00,640 [282] [DEBUG] [app] Ending request: urn:request:b5626a2b-7210-41c6-ba3f-e54d5c62e478 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b5626a2b-7210-41c6-ba3f-e54d5c62e478', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:00,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:35:00,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:00,642 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:00,643 [285] [DEBUG] [app] Starting request: urn:request:c949d786-183b-451c-aff4-b7614e7bbb7a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:00,644 [285] [DEBUG] [app] Ending request: urn:request:c949d786-183b-451c-aff4-b7614e7bbb7a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c949d786-183b-451c-aff4-b7614e7bbb7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:35:00,644 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:00,644 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:00,646 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2024, 4, 22, 3, 35, 0, 645550), None, 'quay', 'quay', datetime.datetime(2024, 4, 22, 3, 35, 0, 645575), 'quay', None, datetime.datetime(2024, 4, 21, 3, 35, 0, 645588), datetime.datetime(2024, 4, 15, 3, 35, 0, 645597), None])
gunicorn-web stdout | 2024-04-22 03:35:00,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:00,661 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:00,662 [286] [DEBUG] [app] Ending request: urn:request:e0a7aae8-49a9-4fe3-96fc-47e567c744fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0a7aae8-49a9-4fe3-96fc-47e567c744fd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:00,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:00,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:35:00,669 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:00,673 [284] [DEBUG] [app] Ending request: urn:request:42343c8f-fd0e-4b0f-b4b4-904943560034 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:42343c8f-fd0e-4b0f-b4b4-904943560034', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:00,673 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:00,674 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.042 118 0.043)
gcworker stdout | 2024-04-22 03:35:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:35:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:31.842666+00:00 (in 29.999423 seconds)
gcworker stdout | 2024-04-22 03:35:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:31 UTC)" (scheduled at 2024-04-22 03:35:01.842666+00:00)
gcworker stdout | 2024-04-22 03:35:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:35:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:35:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:35:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:09.033206+00:00 (in 6.997107 seconds)
repositorygcworker stdout | 2024-04-22 03:35:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:02 UTC)" (scheduled at 2024-04-22 03:35:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:35:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:35:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:35:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:12.430973+00:00 (in 9.999437 seconds)
notificationworker stdout | 2024-04-22 03:35:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:12 UTC)" (scheduled at 2024-04-22 03:35:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:35:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:35:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 2, 431849), True, datetime.datetime(2024, 4, 22, 3, 35, 2, 431849), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:35:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:35:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:35:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:35:03,254 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:35:04,456 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:35:07,131 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:35:08,342 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:35:08,696 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:35:08,707 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:35:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:35:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:02.035613+00:00 (in 53.001931 seconds)
repositorygcworker stdout | 2024-04-22 03:35:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:09 UTC)" (scheduled at 2024-04-22 03:35:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:35:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:35:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 9, 34122), True, datetime.datetime(2024, 4, 22, 3, 35, 9, 34122), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:35:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:35:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:35:09,050 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:35:09,347 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:35:10,021 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:35:10,027 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
autopruneworker stdout | 2024-04-22 03:35:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:35:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:40.238173+00:00 (in 29.999457 seconds)
autopruneworker stdout | 2024-04-22 03:35:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:40 UTC)" (scheduled at 2024-04-22 03:35:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:35:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756850248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:35:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:35:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:47.245913+00:00 (in 36.996775 seconds)
namespacegcworker stdout | 2024-04-22 03:35:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:10 UTC)" (scheduled at 2024-04-22 03:35:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:35:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:35:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:35:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:35:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:35:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:40 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:35:11,426 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:35:12,278 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:35:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:22.430973+00:00 (in 9.999479 seconds)
notificationworker stdout | 2024-04-22 03:35:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:22 UTC)" (scheduled at 2024-04-22 03:35:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:35:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:35:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 12, 431752), True, datetime.datetime(2024, 4, 22, 3, 35, 12, 431752), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:35:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:35:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:35:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:22 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:35:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:35:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:17.065745+00:00 (in 2.998151 seconds)
securityworker stdout | 2024-04-22 03:35:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:44 UTC)" (scheduled at 2024-04-22 03:35:14.067050+00:00)
securityworker stdout | 2024-04-22 03:35:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:35:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:35:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:35:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:35:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:35:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:35:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:35:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:35:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:35:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:35:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 30, 14, 76399), 1, 2])
securityworker stdout | 2024-04-22 03:35:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:35:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:35:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 30, 14, 76399), 1, 2])
securityworker stdout | 2024-04-22 03:35:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:35:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:35:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:35:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:35:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:35:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:35:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:35:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:35:14,727 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:35:15,039 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:35:15,432 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:35:15,633 [282] [DEBUG] [app] Starting request: urn:request:ff748cd0-5456-4455-b7fc-5933a32231f9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:15,633 [285] [DEBUG] [app] Starting request: urn:request:964327b8-2cc3-4602-840f-32285ab15211 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:15,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:15,635 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:15,638 [275] [DEBUG] [app] Starting request: urn:request:cd986f48-078c-4eb8-b66d-7ffd6f7911b8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:35:15,638 [275] [DEBUG] [app] Ending request: urn:request:cd986f48-078c-4eb8-b66d-7ffd6f7911b8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd986f48-078c-4eb8-b66d-7ffd6f7911b8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:35:15,639 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.002)
gunicorn-web stdout | 2024-04-22 03:35:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:35:15,640 [283] [DEBUG] [app] Starting request: urn:request:f87d96cc-f052-490a-8a56-10e90de0afd1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:15,641 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:15,641 [283] [DEBUG] [app] Ending request: urn:request:f87d96cc-f052-490a-8a56-10e90de0afd1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f87d96cc-f052-490a-8a56-10e90de0afd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:15,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:35:15,643 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.006 162 0.006)
gunicorn-web stdout | 2024-04-22 03:35:15,643 [286] [DEBUG] [app] Starting request: urn:request:3d3f32a9-f3d1-478b-bdfe-0dc60a2f6958 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:15,644 [286] [DEBUG] [app] Ending request: urn:request:3d3f32a9-f3d1-478b-bdfe-0dc60a2f6958 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d3f32a9-f3d1-478b-bdfe-0dc60a2f6958', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:15,644 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:15,645 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:15,645 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.002)
gunicorn-web stdout | 2024-04-22 03:35:15,646 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:15,647 [285] [DEBUG] [app] Starting request: urn:request:f7fcd1bc-9959-4064-860f-89af638fd413 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:15,647 [285] [DEBUG] [app] Ending request: urn:request:f7fcd1bc-9959-4064-860f-89af638fd413 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7fcd1bc-9959-4064-860f-89af638fd413', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:15,647 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:15,648 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:35:15,649 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:15,656 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:15,658 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:15,670 [282] [DEBUG] [app] Ending request: urn:request:ff748cd0-5456-4455-b7fc-5933a32231f9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ff748cd0-5456-4455-b7fc-5933a32231f9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:15,670 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:15,671 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:35:15,671 [285] [DEBUG] [app] Ending request: urn:request:964327b8-2cc3-4602-840f-32285ab15211 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:964327b8-2cc3-4602-840f-32285ab15211', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.039)
gunicorn-web stdout | 2024-04-22 03:35:15,671 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:15,672 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.041 118 0.040)
exportactionlogsworker stdout | 2024-04-22 03:35:16,212 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:35:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:35:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:44.067050+00:00 (in 27.000643 seconds)
securityworker stdout | 2024-04-22 03:35:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:47 UTC)" (scheduled at 2024-04-22 03:35:17.065745+00:00)
securityworker stdout | 2024-04-22 03:35:17,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:35:17,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:35:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:35:17,072 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:35:17,085 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:35:17,088 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:35:17,089 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:35:17,312 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:35:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:35:18,048 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:53.045554+00:00 (in 34.997012 seconds)
exportactionlogsworker stdout | 2024-04-22 03:35:18,049 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:18 UTC)" (scheduled at 2024-04-22 03:35:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:35:18,049 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:35:18,049 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:35:18,139 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:35:18,638 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:35:19,277 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:35:19,280 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:35:19,284 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:35:19,752 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:35:20,207 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:35:20,321 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:35:20,690 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:35:20,759 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:35:20,762 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:35:20,768 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:35:20,770 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:35:20,774 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:35:20,777 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:35:20,779 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:35:20,783 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:35:20,978 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:35:20,981 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:35:20,986 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:35:21,001 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:35:21,005 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:35:21,054 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:35:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:32.430973+00:00 (in 9.999433 seconds)
notificationworker stdout | 2024-04-22 03:35:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:32 UTC)" (scheduled at 2024-04-22 03:35:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:35:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:35:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 22, 431843), True, datetime.datetime(2024, 4, 22, 3, 35, 22, 431843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:35:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:35:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:35:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:35:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:35:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:24.738365+00:00 (in 59.999475 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:35:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:24 UTC)" (scheduled at 2024-04-22 03:35:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:35:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:35:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:35:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:35:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:35:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:58.130973+00:00 (in 29.999413 seconds)
buildlogsarchiver stdout | 2024-04-22 03:35:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:58 UTC)" (scheduled at 2024-04-22 03:35:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:35:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 35, 28, 132025), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:35:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:35:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:35:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:35:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:31.933721+00:00 (in 2.002153 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:29 UTC)" (scheduled at 2024-04-22 03:35:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 29, 931843), True, datetime.datetime(2024, 4, 22, 3, 35, 29, 931843), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:35:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:35:30,632 [282] [DEBUG] [app] Starting request: urn:request:168d61d6-e7db-4047-b87b-cb1b86e2c007 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:30,632 [286] [DEBUG] [app] Starting request: urn:request:58e75904-acaa-4c3b-85f9-517ce183a797 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:30,636 [275] [DEBUG] [app] Starting request: urn:request:be40e55c-79b2-431b-a3dc-49934c620874 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:35:30,637 [275] [DEBUG] [app] Ending request: urn:request:be40e55c-79b2-431b-a3dc-49934c620874 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be40e55c-79b2-431b-a3dc-49934c620874', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:35:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:35:30,638 [277] [DEBUG] [app] Starting request: urn:request:82764e13-b313-4256-a550-e0a7151989cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:30,639 [277] [DEBUG] [app] Ending request: urn:request:82764e13-b313-4256-a550-e0a7151989cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82764e13-b313-4256-a550-e0a7151989cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.004 162 0.005)
gunicorn-registry stdout | 2024-04-22 03:35:30,640 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:30,640 [286] [DEBUG] [app] Starting request: urn:request:0e723b68-6679-46a2-af07-11f54a1a321f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:30,640 [286] [DEBUG] [app] Ending request: urn:request:0e723b68-6679-46a2-af07-11f54a1a321f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0e723b68-6679-46a2-af07-11f54a1a321f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:30,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:35:30,641 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:30,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:30,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:30,642 [282] [DEBUG] [app] Starting request: urn:request:fba0bd3f-16ec-44e1-98e5-ca3eed24fb4d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:30,643 [282] [DEBUG] [app] Ending request: urn:request:fba0bd3f-16ec-44e1-98e5-ca3eed24fb4d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fba0bd3f-16ec-44e1-98e5-ca3eed24fb4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:30,643 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:30,643 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:35:30,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:30,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:30,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:30,663 [282] [DEBUG] [app] Ending request: urn:request:168d61d6-e7db-4047-b87b-cb1b86e2c007 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:168d61d6-e7db-4047-b87b-cb1b86e2c007', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:30,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:30,663 [286] [DEBUG] [app] Ending request: urn:request:58e75904-acaa-4c3b-85f9-517ce183a797 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:58e75904-acaa-4c3b-85f9-517ce183a797', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:30,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:35:30,664 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:35:30,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gcworker stdout | 2024-04-22 03:35:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:35:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:01.842666+00:00 (in 29.999423 seconds)
gcworker stdout | 2024-04-22 03:35:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:01 UTC)" (scheduled at 2024-04-22 03:35:31.842666+00:00)
gcworker stdout | 2024-04-22 03:35:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:35:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:35:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:35:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:29.931002+00:00 (in 57.996760 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:35:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:31 UTC)" (scheduled at 2024-04-22 03:35:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:35:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:35:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:35:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:42.430973+00:00 (in 9.999505 seconds)
notificationworker stdout | 2024-04-22 03:35:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:42 UTC)" (scheduled at 2024-04-22 03:35:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:35:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:35:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 32, 431716), True, datetime.datetime(2024, 4, 22, 3, 35, 32, 431716), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:35:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:35:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:35:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:35:33,291 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:35:34,494 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:35:37,169 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:35:38,382 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:35:38,705 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:35:38,745 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:35:39,386 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:35:40,060 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:35:40,066 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
autopruneworker stdout | 2024-04-22 03:35:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:35:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:10.238173+00:00 (in 29.999476 seconds)
autopruneworker stdout | 2024-04-22 03:35:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:10 UTC)" (scheduled at 2024-04-22 03:35:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:35:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756880247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:35:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:35:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:35:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:10 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:35:41,464 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:35:42,317 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:35:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:52.430973+00:00 (in 9.999431 seconds)
notificationworker stdout | 2024-04-22 03:35:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:52 UTC)" (scheduled at 2024-04-22 03:35:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:35:42,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:35:42,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 42, 432062), True, datetime.datetime(2024, 4, 22, 3, 35, 42, 432062), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:35:42,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:35:42,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:35:42,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:35:52 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:35:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:35:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:47.065745+00:00 (in 2.997983 seconds)
securityworker stdout | 2024-04-22 03:35:44,068 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:14 UTC)" (scheduled at 2024-04-22 03:35:44.067050+00:00)
securityworker stdout | 2024-04-22 03:35:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:35:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:35:44,073 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:35:44,073 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:44,073 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:35:44,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:35:44,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:35:44,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:35:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:35:44,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:35:44,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:35:44,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 30, 44, 78850), 1, 2])
securityworker stdout | 2024-04-22 03:35:44,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:35:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:35:44,103 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 30, 44, 78850), 1, 2])
securityworker stdout | 2024-04-22 03:35:44,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:35:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:35:44,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:44,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:35:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:35:44,108 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:35:44,108 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:35:44,109 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:35:44,109 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:35:44,109 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:35:44,768 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:35:45,078 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:35:45,470 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:35:45,632 [286] [DEBUG] [app] Starting request: urn:request:a47c2bed-de6c-4fe4-8b95-2ac94ea91a3d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:45,632 [285] [DEBUG] [app] Starting request: urn:request:5e377ffd-3b13-4103-b7d9-82715c02874e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:35:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:35:45,636 [281] [DEBUG] [app] Starting request: urn:request:e0fce0bc-2c90-4f46-b0dc-96e9d5d65ff4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:35:45,637 [281] [DEBUG] [app] Ending request: urn:request:e0fce0bc-2c90-4f46-b0dc-96e9d5d65ff4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e0fce0bc-2c90-4f46-b0dc-96e9d5d65ff4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:35:45,638 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:35:45,638 [278] [DEBUG] [app] Starting request: urn:request:1aaf3eff-da52-49bc-a0fe-f9e165288f45 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.003)
gunicorn-web stdout | 2024-04-22 03:35:45,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:35:45,639 [278] [DEBUG] [app] Ending request: urn:request:1aaf3eff-da52-49bc-a0fe-f9e165288f45 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1aaf3eff-da52-49bc-a0fe-f9e165288f45', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:45,640 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.005 162 0.004)
gunicorn-registry stdout | 2024-04-22 03:35:45,640 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:45,642 [286] [DEBUG] [app] Starting request: urn:request:ef5849e2-fe2b-4464-8383-a960ea71c2e6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:45,642 [286] [DEBUG] [app] Ending request: urn:request:ef5849e2-fe2b-4464-8383-a960ea71c2e6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef5849e2-fe2b-4464-8383-a960ea71c2e6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:35:45,642 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:35:45,642 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:35:45,643 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:45,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:45,644 [285] [DEBUG] [app] Starting request: urn:request:f80ffd2d-1075-41b1-bedc-ae21099ee8a4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:35:45,645 [285] [DEBUG] [app] Ending request: urn:request:f80ffd2d-1075-41b1-bedc-ae21099ee8a4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f80ffd2d-1075-41b1-bedc-ae21099ee8a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:35:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:35:45,645 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:35:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:35:45,645 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:35:45,646 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:35:45,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:45,654 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:35:45,664 [286] [DEBUG] [app] Ending request: urn:request:a47c2bed-de6c-4fe4-8b95-2ac94ea91a3d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a47c2bed-de6c-4fe4-8b95-2ac94ea91a3d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:45,665 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:45,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:35:45,666 [285] [DEBUG] [app] Ending request: urn:request:5e377ffd-3b13-4103-b7d9-82715c02874e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5e377ffd-3b13-4103-b7d9-82715c02874e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:35:45,666 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:35:45,667 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:35:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:35:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.036)
exportactionlogsworker stdout | 2024-04-22 03:35:46,250 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:35:47,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:35:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:14.067050+00:00 (in 27.000660 seconds)
securityworker stdout | 2024-04-22 03:35:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:17 UTC)" (scheduled at 2024-04-22 03:35:47.065745+00:00)
securityworker stdout | 2024-04-22 03:35:47,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:35:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:35:47,068 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:35:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:35:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:35:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:35:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:35:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:35:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:10.248668+00:00 (in 23.002159 seconds)
namespacegcworker stdout | 2024-04-22 03:35:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:47 UTC)" (scheduled at 2024-04-22 03:35:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:35:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:35:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 47, 246922), True, datetime.datetime(2024, 4, 22, 3, 35, 47, 246922), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:35:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:35:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:35:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:35:47,349 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:35:48,178 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:35:48,675 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:35:49,286 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:35:49,291 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:35:49,294 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:35:49,770 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:35:50,245 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:35:50,358 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:35:50,702 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:35:50,772 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:35:50,775 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:35:50,780 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:35:50,783 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:35:50,787 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:35:50,790 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:35:50,794 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:35:50,797 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:35:50,989 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:35:50,994 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:35:50,998 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:35:51,011 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:35:51,015 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:35:51,092 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:35:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:35:54.433683+00:00 (in 2.002082 seconds)
notificationworker stdout | 2024-04-22 03:35:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:02 UTC)" (scheduled at 2024-04-22 03:35:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:35:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:35:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 52, 431941), True, datetime.datetime(2024, 4, 22, 3, 35, 52, 431941), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:35:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:35:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:35:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:35:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:35:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:18.047111+00:00 (in 25.000957 seconds)
exportactionlogsworker stdout | 2024-04-22 03:35:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:53 UTC)" (scheduled at 2024-04-22 03:35:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:35:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:35:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 35, 53, 46632), True, datetime.datetime(2024, 4, 22, 3, 35, 53, 46632), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:35:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:35:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:35:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:35:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:35:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:02.430973+00:00 (in 7.996652 seconds)
notificationworker stdout | 2024-04-22 03:35:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:54 UTC)" (scheduled at 2024-04-22 03:35:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:35:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:35:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:36:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:35:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:35:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:28.130973+00:00 (in 29.999401 seconds)
buildlogsarchiver stdout | 2024-04-22 03:35:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:28 UTC)" (scheduled at 2024-04-22 03:35:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:35:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 35, 58, 132040), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:35:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:35:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:35:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:36:00,632 [282] [DEBUG] [app] Starting request: urn:request:6cb4f113-b1f6-43c5-9295-618ca9eb635b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:00,633 [286] [DEBUG] [app] Starting request: urn:request:97c2bef7-3fc6-4ce4-8cac-5bbcd79de4cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:00,635 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:00,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:36:00,637 [283] [DEBUG] [app] Starting request: urn:request:bec2acaf-53ac-4e48-bf7b-6dc79bf82c91 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:00,637 [275] [DEBUG] [app] Starting request: urn:request:10ff6b33-b83e-4953-925e-723f5e34073d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:00,637 [283] [DEBUG] [app] Ending request: urn:request:bec2acaf-53ac-4e48-bf7b-6dc79bf82c91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bec2acaf-53ac-4e48-bf7b-6dc79bf82c91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:36:00,637 [275] [DEBUG] [app] Ending request: urn:request:10ff6b33-b83e-4953-925e-723f5e34073d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:10ff6b33-b83e-4953-925e-723f5e34073d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:36:00,638 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:36:00,638 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:36:00,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:00,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:00,640 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:00,640 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:00,641 [286] [DEBUG] [app] Starting request: urn:request:267488ee-de61-43ef-8dd1-d24c6603be93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:00,641 [286] [DEBUG] [app] Ending request: urn:request:267488ee-de61-43ef-8dd1-d24c6603be93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:267488ee-de61-43ef-8dd1-d24c6603be93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:00,642 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:36:00,642 [285] [DEBUG] [app] Starting request: urn:request:c45ac11f-0af9-464b-b6f6-cfe5f964096d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:00,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:00,642 [285] [DEBUG] [app] Ending request: urn:request:c45ac11f-0af9-464b-b6f6-cfe5f964096d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c45ac11f-0af9-464b-b6f6-cfe5f964096d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:00,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:00,643 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:36:00,643 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:00,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:00,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:00,652 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:00,663 [286] [DEBUG] [app] Ending request: urn:request:97c2bef7-3fc6-4ce4-8cac-5bbcd79de4cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:97c2bef7-3fc6-4ce4-8cac-5bbcd79de4cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:00,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:00,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.032)
gunicorn-web stdout | 2024-04-22 03:36:00,665 [282] [DEBUG] [app] Ending request: urn:request:6cb4f113-b1f6-43c5-9295-618ca9eb635b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cb4f113-b1f6-43c5-9295-618ca9eb635b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:00,665 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:00,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.033)
gcworker stdout | 2024-04-22 03:36:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:36:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:31.842666+00:00 (in 29.999422 seconds)
gcworker stdout | 2024-04-22 03:36:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:31 UTC)" (scheduled at 2024-04-22 03:36:01.842666+00:00)
gcworker stdout | 2024-04-22 03:36:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:36:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:36:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:36:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:09.033206+00:00 (in 6.997078 seconds)
repositorygcworker stdout | 2024-04-22 03:36:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:02 UTC)" (scheduled at 2024-04-22 03:36:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:36:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:36:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:36:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:12.430973+00:00 (in 9.999483 seconds)
notificationworker stdout | 2024-04-22 03:36:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:12 UTC)" (scheduled at 2024-04-22 03:36:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:36:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:36:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 2, 431797), True, datetime.datetime(2024, 4, 22, 3, 36, 2, 431797), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:36:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:36:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:36:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:36:03,326 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:36:04,531 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:36:07,200 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:36:08,418 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:36:08,742 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:36:08,764 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:36:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:36:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:02.035613+00:00 (in 53.001880 seconds)
repositorygcworker stdout | 2024-04-22 03:36:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:09 UTC)" (scheduled at 2024-04-22 03:36:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:36:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:36:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 9, 34036), True, datetime.datetime(2024, 4, 22, 3, 36, 9, 34036), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:36:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:36:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:36:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:36:09,420 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:36:10,085 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:36:10,101 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
autopruneworker stdout | 2024-04-22 03:36:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:36:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:40.238173+00:00 (in 29.999417 seconds)
autopruneworker stdout | 2024-04-22 03:36:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:40 UTC)" (scheduled at 2024-04-22 03:36:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:36:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756910247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:36:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:36:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:30.247852+00:00 (in 19.998566 seconds)
namespacegcworker stdout | 2024-04-22 03:36:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:10 UTC)" (scheduled at 2024-04-22 03:36:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:36:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:36:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:36:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:36:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:36:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:40 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:36:11,480 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:36:12,353 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:36:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:22.430973+00:00 (in 9.999455 seconds)
notificationworker stdout | 2024-04-22 03:36:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:22 UTC)" (scheduled at 2024-04-22 03:36:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:36:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:36:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 12, 431833), True, datetime.datetime(2024, 4, 22, 3, 36, 12, 431833), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:36:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:36:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:36:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:22 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:36:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:36:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:17.065745+00:00 (in 2.998050 seconds)
securityworker stdout | 2024-04-22 03:36:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:44 UTC)" (scheduled at 2024-04-22 03:36:14.067050+00:00)
securityworker stdout | 2024-04-22 03:36:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:36:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:36:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:36:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:36:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:36:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:36:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:36:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:36:14,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:36:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:36:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 31, 14, 77028), 1, 2])
securityworker stdout | 2024-04-22 03:36:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:36:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:36:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 31, 14, 77028), 1, 2])
securityworker stdout | 2024-04-22 03:36:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:36:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:36:14,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:14,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:36:14,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:36:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:36:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:36:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:36:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:36:14,805 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:36:15,103 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:36:15,507 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:36:15,632 [282] [DEBUG] [app] Starting request: urn:request:fc26c0fd-b673-410a-a173-0e3ef0542ea9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:15,633 [285] [DEBUG] [app] Starting request: urn:request:46af012c-72a8-4366-be7c-d8a3aff96701 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:36:15,636 [281] [DEBUG] [app] Starting request: urn:request:0f662319-2c68-49a5-bdd4-2e2239938d66 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:15,636 [281] [DEBUG] [app] Ending request: urn:request:0f662319-2c68-49a5-bdd4-2e2239938d66 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f662319-2c68-49a5-bdd4-2e2239938d66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:36:15,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:36:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:36:15,637 [279] [DEBUG] [app] Starting request: urn:request:096ad1a1-a7b0-48c1-853b-78a9cd1e4e5c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:36:15,638 [279] [DEBUG] [app] Ending request: urn:request:096ad1a1-a7b0-48c1-853b-78a9cd1e4e5c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:096ad1a1-a7b0-48c1-853b-78a9cd1e4e5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.004 162 0.004)
gunicorn-registry stdout | 2024-04-22 03:36:15,639 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:15,639 [282] [DEBUG] [app] Starting request: urn:request:e80e243d-06b8-4800-af66-bf54488fe6d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:15,640 [282] [DEBUG] [app] Ending request: urn:request:e80e243d-06b8-4800-af66-bf54488fe6d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e80e243d-06b8-4800-af66-bf54488fe6d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:36:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:15,642 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:15,643 [282] [DEBUG] [app] Starting request: urn:request:3ba1aab7-0a21-4ed9-a809-f2627a7453d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:15,644 [282] [DEBUG] [app] Ending request: urn:request:3ba1aab7-0a21-4ed9-a809-f2627a7453d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ba1aab7-0a21-4ed9-a809-f2627a7453d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:15,644 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:36:15,644 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:15,645 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:15,653 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:15,660 [285] [DEBUG] [app] Ending request: urn:request:46af012c-72a8-4366-be7c-d8a3aff96701 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:46af012c-72a8-4366-be7c-d8a3aff96701', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:15,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:15,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:36:15,664 [282] [DEBUG] [app] Ending request: urn:request:fc26c0fd-b673-410a-a173-0e3ef0542ea9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fc26c0fd-b673-410a-a173-0e3ef0542ea9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:15,665 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:15,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
exportactionlogsworker stdout | 2024-04-22 03:36:16,269 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:36:17,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:36:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:44.067050+00:00 (in 27.000647 seconds)
securityworker stdout | 2024-04-22 03:36:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:47 UTC)" (scheduled at 2024-04-22 03:36:17.065745+00:00)
securityworker stdout | 2024-04-22 03:36:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:36:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:36:17,068 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:36:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:36:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:36:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:36:17,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:36:17,386 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:36:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:36:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:53.045554+00:00 (in 34.997837 seconds)
exportactionlogsworker stdout | 2024-04-22 03:36:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:18 UTC)" (scheduled at 2024-04-22 03:36:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:36:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:36:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:36:18,216 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:36:18,712 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:36:19,294 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:36:19,299 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:36:19,301 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:36:19,807 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:36:20,265 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:36:20,369 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:36:20,712 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:36:20,783 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:36:20,787 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:36:20,790 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:36:20,794 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:36:20,797 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:36:20,799 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:36:20,803 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:36:20,806 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:36:20,998 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:36:21,003 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:36:21,006 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:36:21,019 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:36:21,021 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:36:21,118 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:36:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:32.430973+00:00 (in 9.999441 seconds)
notificationworker stdout | 2024-04-22 03:36:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:32 UTC)" (scheduled at 2024-04-22 03:36:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:36:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:36:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 22, 431838), True, datetime.datetime(2024, 4, 22, 3, 36, 22, 431838), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:36:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:36:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:36:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:36:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:36:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:24.738365+00:00 (in 59.999455 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:36:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:24 UTC)" (scheduled at 2024-04-22 03:36:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:36:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:36:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:36:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:36:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:36:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:58.130973+00:00 (in 29.999420 seconds)
buildlogsarchiver stdout | 2024-04-22 03:36:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:58 UTC)" (scheduled at 2024-04-22 03:36:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:36:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 36, 28, 131921), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:36:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:36:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:36:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:36:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:31.933721+00:00 (in 2.002187 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:29 UTC)" (scheduled at 2024-04-22 03:36:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 29, 931808), True, datetime.datetime(2024, 4, 22, 3, 36, 29, 931808), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:36:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:36:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:36:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:47.245913+00:00 (in 16.997411 seconds)
namespacegcworker stdout | 2024-04-22 03:36:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:41:30 UTC)" (scheduled at 2024-04-22 03:36:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 03:36:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 36, 30, 248917), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:36:30,263 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 36, 30, 248917), True, datetime.datetime(2024, 4, 22, 3, 36, 30, 248917), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:36:30,268 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 36, 30, 248917), True, datetime.datetime(2024, 4, 22, 3, 36, 30, 248917), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 3, 36, 30, 248917), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:36:30,272 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:36:30,272 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:41:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:36:30,632 [282] [DEBUG] [app] Starting request: urn:request:d8ed641d-dfa9-477c-b8a4-2d17132ae00a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:30,632 [286] [DEBUG] [app] Starting request: urn:request:1e530768-a8c4-49d2-b63a-30faad3d049f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:36:30,636 [275] [DEBUG] [app] Starting request: urn:request:3c61d969-7227-4582-bbb2-2a56b341f3d0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:30,636 [275] [DEBUG] [app] Ending request: urn:request:3c61d969-7227-4582-bbb2-2a56b341f3d0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c61d969-7227-4582-bbb2-2a56b341f3d0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:36:30,637 [283] [DEBUG] [app] Starting request: urn:request:401ef2ec-9126-4d4d-ab4c-fb18b74b95af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:36:30,637 [283] [DEBUG] [app] Ending request: urn:request:401ef2ec-9126-4d4d-ab4c-fb18b74b95af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:401ef2ec-9126-4d4d-ab4c-fb18b74b95af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:36:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:36:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:30,639 [286] [DEBUG] [app] Starting request: urn:request:bb67d203-66f7-42f1-831a-62cb7875e3fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:30,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:30,639 [286] [DEBUG] [app] Ending request: urn:request:bb67d203-66f7-42f1-831a-62cb7875e3fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb67d203-66f7-42f1-831a-62cb7875e3fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:36:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:30,641 [285] [DEBUG] [app] Starting request: urn:request:e9d3b59b-41c3-417c-a999-1274dc835913 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:30,641 [285] [DEBUG] [app] Ending request: urn:request:e9d3b59b-41c3-417c-a999-1274dc835913 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9d3b59b-41c3-417c-a999-1274dc835913', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:36:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:30,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:36:30,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:30,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:30,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:30,663 [282] [DEBUG] [app] Ending request: urn:request:d8ed641d-dfa9-477c-b8a4-2d17132ae00a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d8ed641d-dfa9-477c-b8a4-2d17132ae00a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:30,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:30,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:36:30,670 [286] [DEBUG] [app] Ending request: urn:request:1e530768-a8c4-49d2-b63a-30faad3d049f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1e530768-a8c4-49d2-b63a-30faad3d049f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:30,670 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:30,670 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.039)
gcworker stdout | 2024-04-22 03:36:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:36:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:01.842666+00:00 (in 29.999411 seconds)
gcworker stdout | 2024-04-22 03:36:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:01 UTC)" (scheduled at 2024-04-22 03:36:31.842666+00:00)
gcworker stdout | 2024-04-22 03:36:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:36:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:36:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:36:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:29.931002+00:00 (in 57.996685 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:36:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:31 UTC)" (scheduled at 2024-04-22 03:36:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:36:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:36:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:36:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:42.430973+00:00 (in 9.999485 seconds)
notificationworker stdout | 2024-04-22 03:36:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:42 UTC)" (scheduled at 2024-04-22 03:36:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:36:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:36:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 32, 431753), True, datetime.datetime(2024, 4, 22, 3, 36, 32, 431753), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:36:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:36:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:36:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:36:33,355 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:36:34,567 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:36:37,234 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:36:38,457 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:36:38,779 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:36:38,790 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:36:39,457 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:36:40,122 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:36:40,136 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
autopruneworker stdout | 2024-04-22 03:36:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:36:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:10.238173+00:00 (in 29.999541 seconds)
autopruneworker stdout | 2024-04-22 03:36:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:10 UTC)" (scheduled at 2024-04-22 03:36:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:36:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756940247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:36:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:36:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:36:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:10 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:36:41,518 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:36:42,381 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:36:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:52.430973+00:00 (in 9.999465 seconds)
notificationworker stdout | 2024-04-22 03:36:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:52 UTC)" (scheduled at 2024-04-22 03:36:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:36:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:36:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 42, 431795), True, datetime.datetime(2024, 4, 22, 3, 36, 42, 431795), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:36:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:36:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:36:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:36:52 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:36:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:36:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:47.065745+00:00 (in 2.998072 seconds)
securityworker stdout | 2024-04-22 03:36:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:14 UTC)" (scheduled at 2024-04-22 03:36:44.067050+00:00)
securityworker stdout | 2024-04-22 03:36:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:36:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:36:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:36:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:36:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:36:44,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:36:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:36:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:36:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:36:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:36:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 31, 44, 76781), 1, 2])
securityworker stdout | 2024-04-22 03:36:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:36:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:36:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 31, 44, 76781), 1, 2])
securityworker stdout | 2024-04-22 03:36:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:36:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:36:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:36:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:36:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:36:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:36:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:36:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:36:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:36:44,842 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:36:45,140 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:36:45,520 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:36:45,632 [286] [DEBUG] [app] Starting request: urn:request:e710e8fb-f8b3-4e35-8f41-b1c0fe4eb7d3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:45,632 [282] [DEBUG] [app] Starting request: urn:request:d198c142-f32a-40b7-af9a-26934c86e74c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:36:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:36:45,636 [275] [DEBUG] [app] Starting request: urn:request:9aec2482-d293-4a0a-a07c-eb33823121dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:45,636 [279] [DEBUG] [app] Starting request: urn:request:082998b9-7f50-44b3-b1b5-28f37f3caade (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:36:45,636 [275] [DEBUG] [app] Ending request: urn:request:9aec2482-d293-4a0a-a07c-eb33823121dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9aec2482-d293-4a0a-a07c-eb33823121dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:36:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:36:45,636 [279] [DEBUG] [app] Ending request: urn:request:082998b9-7f50-44b3-b1b5-28f37f3caade (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:082998b9-7f50-44b3-b1b5-28f37f3caade', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:36:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:36:45,637 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:36:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:45,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:36:45,639 [286] [DEBUG] [app] Starting request: urn:request:16835cfa-0c87-4583-8243-54f687f87059 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:45,639 [286] [DEBUG] [app] Ending request: urn:request:16835cfa-0c87-4583-8243-54f687f87059 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:16835cfa-0c87-4583-8243-54f687f87059', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:45,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:36:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:45,641 [285] [DEBUG] [app] Starting request: urn:request:561b22d3-bd1e-4d6b-af0e-f4546a1f6e3a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:36:45,641 [285] [DEBUG] [app] Ending request: urn:request:561b22d3-bd1e-4d6b-af0e-f4546a1f6e3a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:561b22d3-bd1e-4d6b-af0e-f4546a1f6e3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:36:45,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:36:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:36:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:36:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:36:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:36:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:36:45,661 [286] [DEBUG] [app] Ending request: urn:request:e710e8fb-f8b3-4e35-8f41-b1c0fe4eb7d3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e710e8fb-f8b3-4e35-8f41-b1c0fe4eb7d3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:36:45,662 [282] [DEBUG] [app] Ending request: urn:request:d198c142-f32a-40b7-af9a-26934c86e74c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d198c142-f32a-40b7-af9a-26934c86e74c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:36:45,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:36:45,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:36:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:36:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
exportactionlogsworker stdout | 2024-04-22 03:36:46,305 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:36:47,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:36:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:14.067050+00:00 (in 27.000693 seconds)
securityworker stdout | 2024-04-22 03:36:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:17 UTC)" (scheduled at 2024-04-22 03:36:47.065745+00:00)
securityworker stdout | 2024-04-22 03:36:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:36:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:36:47,068 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:36:47,069 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:36:47,081 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:36:47,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:36:47,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:36:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:36:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:10.248668+00:00 (in 23.002173 seconds)
namespacegcworker stdout | 2024-04-22 03:36:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:47 UTC)" (scheduled at 2024-04-22 03:36:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:36:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:36:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 47, 246953), True, datetime.datetime(2024, 4, 22, 3, 36, 47, 246953), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:36:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:36:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:36:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:36:47,418 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:36:48,252 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:36:48,748 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:36:49,304 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:36:49,308 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:36:49,312 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:36:49,832 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:36:50,302 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:36:50,397 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:36:50,722 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:36:50,792 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:36:50,796 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:36:50,798 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:36:50,801 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:36:50,805 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:36:50,809 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:36:50,812 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:36:50,815 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:36:51,006 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:36:51,012 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:36:51,014 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:36:51,026 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:36:51,030 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:36:51,154 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:36:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:36:54.433683+00:00 (in 2.002185 seconds)
notificationworker stdout | 2024-04-22 03:36:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:02 UTC)" (scheduled at 2024-04-22 03:36:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:36:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:36:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 52, 431773), True, datetime.datetime(2024, 4, 22, 3, 36, 52, 431773), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:36:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:36:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:36:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:36:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:36:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:18.047111+00:00 (in 25.000930 seconds)
exportactionlogsworker stdout | 2024-04-22 03:36:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:53 UTC)" (scheduled at 2024-04-22 03:36:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:36:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:36:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 36, 53, 46452), True, datetime.datetime(2024, 4, 22, 3, 36, 53, 46452), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:36:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:36:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:36:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:36:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:36:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:02.430973+00:00 (in 7.996793 seconds)
notificationworker stdout | 2024-04-22 03:36:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:54 UTC)" (scheduled at 2024-04-22 03:36:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:36:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:36:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:37:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:36:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:36:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:28.130973+00:00 (in 29.999406 seconds)
buildlogsarchiver stdout | 2024-04-22 03:36:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:28 UTC)" (scheduled at 2024-04-22 03:36:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:36:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 36, 58, 131953), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:36:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:36:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:36:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:37:00,633 [286] [DEBUG] [app] Starting request: urn:request:2a898c85-2b00-4642-9474-e5112b2c1b11 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:00,633 [282] [DEBUG] [app] Starting request: urn:request:e0d15920-5dc0-4305-addf-4699bdea8948 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:37:00,636 [275] [DEBUG] [app] Starting request: urn:request:161aa6b4-c393-4dfe-920b-1423dc3e0e91 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:00,636 [281] [DEBUG] [app] Starting request: urn:request:6c40604c-5c2f-44b0-b14d-b363f68e48a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:00,636 [281] [DEBUG] [app] Ending request: urn:request:6c40604c-5c2f-44b0-b14d-b363f68e48a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c40604c-5c2f-44b0-b14d-b363f68e48a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:37:00,636 [275] [DEBUG] [app] Ending request: urn:request:161aa6b4-c393-4dfe-920b-1423dc3e0e91 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:161aa6b4-c393-4dfe-920b-1423dc3e0e91', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:37:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:37:00,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:37:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:00,640 [285] [DEBUG] [app] Starting request: urn:request:9a6c3ac6-3d12-4ef9-8f90-0b898a9e5ba1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:00,640 [284] [DEBUG] [app] Starting request: urn:request:c9a2d931-f5c0-4e50-b92b-6baaf4cc0099 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:00,641 [285] [DEBUG] [app] Ending request: urn:request:9a6c3ac6-3d12-4ef9-8f90-0b898a9e5ba1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9a6c3ac6-3d12-4ef9-8f90-0b898a9e5ba1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:00,641 [284] [DEBUG] [app] Ending request: urn:request:c9a2d931-f5c0-4e50-b92b-6baaf4cc0099 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9a2d931-f5c0-4e50-b92b-6baaf4cc0099', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:37:00,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:00,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:00,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:00,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:00,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:00,662 [286] [DEBUG] [app] Ending request: urn:request:2a898c85-2b00-4642-9474-e5112b2c1b11 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a898c85-2b00-4642-9474-e5112b2c1b11', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:37:00,663 [282] [DEBUG] [app] Ending request: urn:request:e0d15920-5dc0-4305-addf-4699bdea8948 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0d15920-5dc0-4305-addf-4699bdea8948', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:37:00,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:00,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 03:37:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:37:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:31.842666+00:00 (in 29.999492 seconds)
gcworker stdout | 2024-04-22 03:37:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:31 UTC)" (scheduled at 2024-04-22 03:37:01.842666+00:00)
gcworker stdout | 2024-04-22 03:37:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:37:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:37:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:37:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:09.033206+00:00 (in 6.997036 seconds)
repositorygcworker stdout | 2024-04-22 03:37:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:02 UTC)" (scheduled at 2024-04-22 03:37:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:37:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:37:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:37:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:12.430973+00:00 (in 9.999388 seconds)
notificationworker stdout | 2024-04-22 03:37:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:12 UTC)" (scheduled at 2024-04-22 03:37:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:37:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:37:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 2, 431863), True, datetime.datetime(2024, 4, 22, 3, 37, 2, 431863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:37:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:37:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:37:03,375 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:37:04,604 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:37:07,255 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:37:08,493 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:37:08,802 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:37:08,821 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:37:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:37:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:02.035613+00:00 (in 53.001912 seconds)
repositorygcworker stdout | 2024-04-22 03:37:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:09 UTC)" (scheduled at 2024-04-22 03:37:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:37:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:37:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 9, 33988), True, datetime.datetime(2024, 4, 22, 3, 37, 9, 33988), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:37:09,057 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:37:09,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:37:09,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:37:09,494 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
storagereplication stdout | 2024-04-22 03:37:10,157 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
teamsyncworker stdout | 2024-04-22 03:37:10,160 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
autopruneworker stdout | 2024-04-22 03:37:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:37:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:40.238173+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2024-04-22 03:37:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:40 UTC)" (scheduled at 2024-04-22 03:37:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:37:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713756970247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:37:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:37:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:47.245913+00:00 (in 36.996712 seconds)
namespacegcworker stdout | 2024-04-22 03:37:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:10 UTC)" (scheduled at 2024-04-22 03:37:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:37:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:37:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:37:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:37:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:37:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:40 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:37:11,555 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:37:12,418 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
notificationworker stdout | 2024-04-22 03:37:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:22.430973+00:00 (in 9.999459 seconds)
notificationworker stdout | 2024-04-22 03:37:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:22 UTC)" (scheduled at 2024-04-22 03:37:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:37:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:37:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 12, 431753), True, datetime.datetime(2024, 4, 22, 3, 37, 12, 431753), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:37:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:37:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:22 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:37:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:37:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:17.065745+00:00 (in 2.998128 seconds)
securityworker stdout | 2024-04-22 03:37:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:44 UTC)" (scheduled at 2024-04-22 03:37:14.067050+00:00)
securityworker stdout | 2024-04-22 03:37:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:37:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:37:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:37:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:37:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:37:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:37:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:37:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:37:14,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:37:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:37:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 32, 14, 76293), 1, 2])
securityworker stdout | 2024-04-22 03:37:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:37:14,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:37:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 32, 14, 76293), 1, 2])
securityworker stdout | 2024-04-22 03:37:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:37:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:37:14,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:14,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:37:14,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:37:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:37:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:37:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:37:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:37:14,866 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:37:15,177 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:37:15,558 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:37:15,632 [286] [DEBUG] [app] Starting request: urn:request:f0f2e28e-8d08-43b6-a96c-53604e78ab59 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:15,632 [282] [DEBUG] [app] Starting request: urn:request:2b323849-7ec0-426b-81d4-5517c6519c4d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:37:15,635 [283] [DEBUG] [app] Starting request: urn:request:1dd0a5e5-7ab2-421e-8e6c-18e22c0cbcf1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:15,635 [283] [DEBUG] [app] Ending request: urn:request:1dd0a5e5-7ab2-421e-8e6c-18e22c0cbcf1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1dd0a5e5-7ab2-421e-8e6c-18e22c0cbcf1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:37:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:37:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:37:15,637 [280] [DEBUG] [app] Starting request: urn:request:993bbb2b-5854-45f4-a2e2-b0a976874a78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:37:15,638 [280] [DEBUG] [app] Ending request: urn:request:993bbb2b-5854-45f4-a2e2-b0a976874a78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:993bbb2b-5854-45f4-a2e2-b0a976874a78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.003)
gunicorn-registry stdout | 2024-04-22 03:37:15,638 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:15,638 [282] [DEBUG] [app] Starting request: urn:request:ad117225-48a6-4238-a3dc-9133c3dfbe6a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:15,639 [282] [DEBUG] [app] Ending request: urn:request:ad117225-48a6-4238-a3dc-9133c3dfbe6a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ad117225-48a6-4238-a3dc-9133c3dfbe6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:37:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:15,641 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:15,643 [285] [DEBUG] [app] Starting request: urn:request:8742716d-ff8f-43a8-a8b2-752994ab2747 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:15,643 [285] [DEBUG] [app] Ending request: urn:request:8742716d-ff8f-43a8-a8b2-752994ab2747 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8742716d-ff8f-43a8-a8b2-752994ab2747', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:15,644 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:15,644 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:37:15,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:15,653 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:15,660 [286] [DEBUG] [app] Ending request: urn:request:f0f2e28e-8d08-43b6-a96c-53604e78ab59 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0f2e28e-8d08-43b6-a96c-53604e78ab59', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:37:15,664 [282] [DEBUG] [app] Ending request: urn:request:2b323849-7ec0-426b-81d4-5517c6519c4d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b323849-7ec0-426b-81d4-5517c6519c4d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:15,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:15,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
exportactionlogsworker stdout | 2024-04-22 03:37:16,342 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:37:17,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:37:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:44.067050+00:00 (in 27.000608 seconds)
securityworker stdout | 2024-04-22 03:37:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:47 UTC)" (scheduled at 2024-04-22 03:37:17.065745+00:00)
securityworker stdout | 2024-04-22 03:37:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:37:17,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:37:17,068 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:37:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:37:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:37:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:37:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:37:17,442 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:37:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:37:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:23.046900+00:00 (in 4.999024 seconds)
exportactionlogsworker stdout | 2024-04-22 03:37:18,048 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:18 UTC)" (scheduled at 2024-04-22 03:37:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:37:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:37:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:37:18,276 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:37:18,786 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:37:19,313 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:37:19,317 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:37:19,321 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:37:19,873 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:37:20,343 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:37:20,437 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:37:20,740 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:37:20,803 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:37:20,812 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:37:20,820 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:37:20,824 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:37:20,828 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:37:20,832 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:37:20,835 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:37:20,839 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:37:21,021 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:37:21,026 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:37:21,029 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:37:21,038 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:37:21,042 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:37:21,169 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:37:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:32.430973+00:00 (in 9.999358 seconds)
notificationworker stdout | 2024-04-22 03:37:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:32 UTC)" (scheduled at 2024-04-22 03:37:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:37:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:37:22,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 22, 431971), True, datetime.datetime(2024, 4, 22, 3, 37, 22, 431971), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:37:22,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:37:22,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:22,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:32 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:37:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:37:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:53.045554+00:00 (in 29.998012 seconds)
exportactionlogsworker stdout | 2024-04-22 03:37:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:42:23 UTC)" (scheduled at 2024-04-22 03:37:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 03:37:23,049 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 37, 23, 48144), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:37:23,064 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 37, 23, 48144), True, datetime.datetime(2024, 4, 22, 3, 37, 23, 48144), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:37:23,069 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 37, 23, 48144), True, datetime.datetime(2024, 4, 22, 3, 37, 23, 48144), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 3, 37, 23, 48144), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:37:23,073 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:37:23,073 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:42:23 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:37:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:37:24,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:24.738365+00:00 (in 59.999249 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:37:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:24 UTC)" (scheduled at 2024-04-22 03:37:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:37:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:37:24,753 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:37:24,754 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:37:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:37:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:58.130973+00:00 (in 29.999388 seconds)
buildlogsarchiver stdout | 2024-04-22 03:37:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:58 UTC)" (scheduled at 2024-04-22 03:37:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:37:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 37, 28, 131859), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:37:28,148 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:37:28,148 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:37:28,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:37:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:31.933721+00:00 (in 2.002121 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:29 UTC)" (scheduled at 2024-04-22 03:37:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,933 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 29, 931951), True, datetime.datetime(2024, 4, 22, 3, 37, 29, 931951), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,948 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,948 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:37:29,948 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:37:30,634 [285] [DEBUG] [app] Starting request: urn:request:d85da125-5520-4007-b2b9-029f0c52b463 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:30,634 [282] [DEBUG] [app] Starting request: urn:request:20ee4ba1-d07d-42fc-bcaf-3b26e8ced158 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:30,636 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:37:30,640 [275] [DEBUG] [app] Starting request: urn:request:5ab5eb51-f027-462b-8c18-bc07d04a2561 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:30,640 [275] [DEBUG] [app] Ending request: urn:request:5ab5eb51-f027-462b-8c18-bc07d04a2561 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5ab5eb51-f027-462b-8c18-bc07d04a2561', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:37:30,641 [283] [DEBUG] [app] Starting request: urn:request:8b839acd-7c35-4b12-958a-f80133f85660 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:30,641 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.003)
gunicorn-registry stdout | 2024-04-22 03:37:30,641 [283] [DEBUG] [app] Ending request: urn:request:8b839acd-7c35-4b12-958a-f80133f85660 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8b839acd-7c35-4b12-958a-f80133f85660', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:30,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:37:30,642 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:30,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:30,644 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:30,645 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:30,646 [285] [DEBUG] [app] Starting request: urn:request:02fda61e-3226-4948-8d05-17143147f5b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:30,647 [285] [DEBUG] [app] Ending request: urn:request:02fda61e-3226-4948-8d05-17143147f5b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02fda61e-3226-4948-8d05-17143147f5b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:30,647 [284] [DEBUG] [app] Starting request: urn:request:9e33b23f-2cf6-4b02-95bf-1a19709b54dc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:30,647 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:37:30,648 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:30,648 [284] [DEBUG] [app] Ending request: urn:request:9e33b23f-2cf6-4b02-95bf-1a19709b54dc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9e33b23f-2cf6-4b02-95bf-1a19709b54dc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:30,648 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:30,648 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:30,649 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:37:30,649 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:30,658 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:30,658 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:30,674 [282] [DEBUG] [app] Ending request: urn:request:20ee4ba1-d07d-42fc-bcaf-3b26e8ced158 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20ee4ba1-d07d-42fc-bcaf-3b26e8ced158', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:30,674 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:30,675 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:37:30,675 [285] [DEBUG] [app] Ending request: urn:request:d85da125-5520-4007-b2b9-029f0c52b463 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d85da125-5520-4007-b2b9-029f0c52b463', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:30,675 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.043 118 0.043)
gunicorn-web stdout | 2024-04-22 03:37:30,676 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.044 118 0.044)
gcworker stdout | 2024-04-22 03:37:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:37:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:01.842666+00:00 (in 29.999248 seconds)
gcworker stdout | 2024-04-22 03:37:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:01 UTC)" (scheduled at 2024-04-22 03:37:31.842666+00:00)
gcworker stdout | 2024-04-22 03:37:31,844 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:37:31,844 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:37:31,934 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:37:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:29.931002+00:00 (in 57.996596 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:37:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:31 UTC)" (scheduled at 2024-04-22 03:37:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:37:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:37:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:37:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:38.433394+00:00 (in 6.001756 seconds)
notificationworker stdout | 2024-04-22 03:37:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:42 UTC)" (scheduled at 2024-04-22 03:37:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:37:32,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:37:32,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 32, 432072), True, datetime.datetime(2024, 4, 22, 3, 37, 32, 432072), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:37:32,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:37:32,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:32,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:37:33,414 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:37:34,642 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:37:37,266 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
notificationworker stdout | 2024-04-22 03:37:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:38,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:42.430973+00:00 (in 3.996982 seconds)
notificationworker stdout | 2024-04-22 03:37:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:42:38 UTC)" (scheduled at 2024-04-22 03:37:38.433394+00:00)
notificationworker stdout | 2024-04-22 03:37:38,435 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 37, 38, 434510), 'notification/%'])
notificationworker stdout | 2024-04-22 03:37:38,450 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 37, 38, 434510), True, datetime.datetime(2024, 4, 22, 3, 37, 38, 434510), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 03:37:38,455 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 37, 38, 434510), True, datetime.datetime(2024, 4, 22, 3, 37, 38, 434510), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 3, 37, 38, 434510), 'notification/%'])
notificationworker stdout | 2024-04-22 03:37:38,459 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:38,459 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:42:38 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:37:38,531 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:37:38,843 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:37:38,847 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:37:39,534 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
storagereplication stdout | 2024-04-22 03:37:40,193 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
teamsyncworker stdout | 2024-04-22 03:37:40,197 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
autopruneworker stdout | 2024-04-22 03:37:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:37:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:10.238173+00:00 (in 29.999492 seconds)
autopruneworker stdout | 2024-04-22 03:37:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:10 UTC)" (scheduled at 2024-04-22 03:37:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:37:40,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757000247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:37:40,255 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:37:40,255 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:37:40,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:10 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:37:41,594 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:37:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:52.430973+00:00 (in 9.999474 seconds)
notificationworker stdout | 2024-04-22 03:37:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:52 UTC)" (scheduled at 2024-04-22 03:37:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:37:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:37:42,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 42, 432028), True, datetime.datetime(2024, 4, 22, 3, 37, 42, 432028), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:37:42,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:37:42,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:42,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:37:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:37:42,458 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:37:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:37:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:47.065745+00:00 (in 2.998104 seconds)
securityworker stdout | 2024-04-22 03:37:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:14 UTC)" (scheduled at 2024-04-22 03:37:44.067050+00:00)
securityworker stdout | 2024-04-22 03:37:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:37:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:37:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:37:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:37:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:37:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:37:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:37:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:37:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:37:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:37:44,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 32, 44, 78235), 1, 2])
securityworker stdout | 2024-04-22 03:37:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:37:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:37:44,103 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 32, 44, 78235), 1, 2])
securityworker stdout | 2024-04-22 03:37:44,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:37:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:37:44,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:44,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:37:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:37:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:37:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:37:44,109 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:37:44,109 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:37:44,109 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:37:44,903 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:37:45,214 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:37:45,591 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:37:45,632 [282] [DEBUG] [app] Starting request: urn:request:ba07a5a7-05e0-48ba-a1df-8ed90840ab55 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:45,632 [285] [DEBUG] [app] Starting request: urn:request:2b5115b7-71ef-4a3b-8c64-ce4bcba2e5e2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:37:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:37:45,636 [281] [DEBUG] [app] Starting request: urn:request:564c0e5a-332d-408a-8221-a77199746985 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:45,636 [280] [DEBUG] [app] Starting request: urn:request:e9d1df40-5087-4867-8fab-217ad495b028 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:37:45,637 [281] [DEBUG] [app] Ending request: urn:request:564c0e5a-332d-408a-8221-a77199746985 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:564c0e5a-332d-408a-8221-a77199746985', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:37:45,637 [280] [DEBUG] [app] Ending request: urn:request:e9d1df40-5087-4867-8fab-217ad495b028 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e9d1df40-5087-4867-8fab-217ad495b028', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:37:45,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:37:45,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:45,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:37:45,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:45,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:37:45,641 [282] [DEBUG] [app] Starting request: urn:request:3801e8a3-efca-4066-b07e-1b7b96b25a86 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:45,641 [285] [DEBUG] [app] Starting request: urn:request:941b39ae-3800-4d4f-be08-9b27ef082d1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:37:45,641 [282] [DEBUG] [app] Ending request: urn:request:3801e8a3-efca-4066-b07e-1b7b96b25a86 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3801e8a3-efca-4066-b07e-1b7b96b25a86', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:45,641 [285] [DEBUG] [app] Ending request: urn:request:941b39ae-3800-4d4f-be08-9b27ef082d1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:941b39ae-3800-4d4f-be08-9b27ef082d1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:37:45,642 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:37:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:37:45,642 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:37:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:37:45,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:45,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:37:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:45,643 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:37:45,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:45,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:37:45,664 [285] [DEBUG] [app] Ending request: urn:request:2b5115b7-71ef-4a3b-8c64-ce4bcba2e5e2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2b5115b7-71ef-4a3b-8c64-ce4bcba2e5e2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:45,664 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:45,664 [282] [DEBUG] [app] Ending request: urn:request:ba07a5a7-05e0-48ba-a1df-8ed90840ab55 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba07a5a7-05e0-48ba-a1df-8ed90840ab55', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:37:45,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:37:45,664 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:37:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:37:45,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:37:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
exportactionlogsworker stdout | 2024-04-22 03:37:46,380 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:37:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:37:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:14.067050+00:00 (in 27.000755 seconds)
securityworker stdout | 2024-04-22 03:37:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:17 UTC)" (scheduled at 2024-04-22 03:37:47.065745+00:00)
securityworker stdout | 2024-04-22 03:37:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:37:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:37:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:37:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:37:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:37:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:37:47,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:37:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:37:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:10.248668+00:00 (in 23.002126 seconds)
namespacegcworker stdout | 2024-04-22 03:37:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:47 UTC)" (scheduled at 2024-04-22 03:37:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:37:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:37:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 47, 246788), True, datetime.datetime(2024, 4, 22, 3, 37, 47, 246788), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:37:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:37:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:37:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:37:47,463 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:37:48,314 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:37:48,825 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:37:49,325 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:37:49,330 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:37:49,336 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:37:49,912 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:37:50,371 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:37:50,476 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:37:50,756 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:37:50,813 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:37:50,823 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:37:50,832 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:37:50,835 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:37:50,841 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:37:50,845 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:37:50,849 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:37:50,852 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:37:51,033 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:37:51,038 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:37:51,042 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:37:51,052 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:37:51,057 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:37:51,209 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:37:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:37:54.433683+00:00 (in 2.002182 seconds)
notificationworker stdout | 2024-04-22 03:37:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:02 UTC)" (scheduled at 2024-04-22 03:37:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:37:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:37:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 52, 431810), True, datetime.datetime(2024, 4, 22, 3, 37, 52, 431810), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:37:52,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:37:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:37:52,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:37:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:37:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:18.047111+00:00 (in 25.000879 seconds)
exportactionlogsworker stdout | 2024-04-22 03:37:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:53 UTC)" (scheduled at 2024-04-22 03:37:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:37:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:37:53,048 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 37, 53, 46764), True, datetime.datetime(2024, 4, 22, 3, 37, 53, 46764), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:37:53,063 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:37:53,063 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:37:53,063 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:37:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:37:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:02.430973+00:00 (in 7.996736 seconds)
notificationworker stdout | 2024-04-22 03:37:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:54 UTC)" (scheduled at 2024-04-22 03:37:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:37:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:37:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:38:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:37:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:37:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:28.130973+00:00 (in 29.999339 seconds)
buildlogsarchiver stdout | 2024-04-22 03:37:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:28 UTC)" (scheduled at 2024-04-22 03:37:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:37:58,133 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 37, 58, 132117), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:37:58,148 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:37:58,149 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:37:58,149 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:38:00,632 [286] [DEBUG] [app] Starting request: urn:request:8220c944-d4b0-4fb5-bebb-ba9b64fcfe0a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:00,632 [282] [DEBUG] [app] Starting request: urn:request:78212b87-ed36-4640-a716-b21d9e700bd0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:38:00,636 [283] [DEBUG] [app] Starting request: urn:request:77bd87f7-8eed-45ec-be99-37f9706b49c3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:00,636 [275] [DEBUG] [app] Starting request: urn:request:11d6381d-60f2-413f-8105-3f02387fd398 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:00,636 [283] [DEBUG] [app] Ending request: urn:request:77bd87f7-8eed-45ec-be99-37f9706b49c3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77bd87f7-8eed-45ec-be99-37f9706b49c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:00,636 [275] [DEBUG] [app] Ending request: urn:request:11d6381d-60f2-413f-8105-3f02387fd398 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11d6381d-60f2-413f-8105-3f02387fd398', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:38:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:00,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:00,641 [286] [DEBUG] [app] Starting request: urn:request:4fe575a2-34d7-4f7b-9c4b-f7010a6e2a20 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:00,641 [284] [DEBUG] [app] Starting request: urn:request:463bffea-79ee-4449-ba0a-6ae5faae7416 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:00,641 [286] [DEBUG] [app] Ending request: urn:request:4fe575a2-34d7-4f7b-9c4b-f7010a6e2a20 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4fe575a2-34d7-4f7b-9c4b-f7010a6e2a20', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:00,641 [284] [DEBUG] [app] Ending request: urn:request:463bffea-79ee-4449-ba0a-6ae5faae7416 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:463bffea-79ee-4449-ba0a-6ae5faae7416', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:00,642 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:00,642 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:38:00,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:00,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:00,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:00,643 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:00,652 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:00,653 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:00,665 [282] [DEBUG] [app] Ending request: urn:request:78212b87-ed36-4640-a716-b21d9e700bd0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78212b87-ed36-4640-a716-b21d9e700bd0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:00,665 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:38:00,665 [286] [DEBUG] [app] Ending request: urn:request:8220c944-d4b0-4fb5-bebb-ba9b64fcfe0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8220c944-d4b0-4fb5-bebb-ba9b64fcfe0a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:00,666 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:38:00,666 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:38:00,666 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.035)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.035)
gcworker stdout | 2024-04-22 03:38:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:38:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:31.842666+00:00 (in 29.999371 seconds)
gcworker stdout | 2024-04-22 03:38:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:31 UTC)" (scheduled at 2024-04-22 03:38:01.842666+00:00)
gcworker stdout | 2024-04-22 03:38:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:38:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:38:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:38:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:09.033206+00:00 (in 6.997015 seconds)
repositorygcworker stdout | 2024-04-22 03:38:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:02 UTC)" (scheduled at 2024-04-22 03:38:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:38:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:38:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:38:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:12.430973+00:00 (in 9.999353 seconds)
notificationworker stdout | 2024-04-22 03:38:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:12 UTC)" (scheduled at 2024-04-22 03:38:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:38:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:38:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 2, 431975), True, datetime.datetime(2024, 4, 22, 3, 38, 2, 431975), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:38:02,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:38:02,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:38:02,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:38:03,452 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:38:04,681 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:38:07,305 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:38:08,569 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:38:08,860 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:38:08,886 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:38:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:38:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:26.035364+00:00 (in 17.001588 seconds)
repositorygcworker stdout | 2024-04-22 03:38:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:09 UTC)" (scheduled at 2024-04-22 03:38:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:38:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:38:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 9, 34106), True, datetime.datetime(2024, 4, 22, 3, 38, 9, 34106), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:38:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:38:09,050 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:38:09,050 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:38:09,574 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
storagereplication stdout | 2024-04-22 03:38:10,231 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
teamsyncworker stdout | 2024-04-22 03:38:10,234 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
autopruneworker stdout | 2024-04-22 03:38:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:38:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:40.238173+00:00 (in 29.999418 seconds)
autopruneworker stdout | 2024-04-22 03:38:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:40 UTC)" (scheduled at 2024-04-22 03:38:10.238173+00:00)
namespacegcworker stdout | 2024-04-22 03:38:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:38:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:47.245913+00:00 (in 36.996691 seconds)
autopruneworker stdout | 2024-04-22 03:38:10,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757030248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:38:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:10 UTC)" (scheduled at 2024-04-22 03:38:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:38:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:38:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:38:10,256 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:38:10,256 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:38:10,256 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:40 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:38:11,633 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:38:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:22.430973+00:00 (in 9.999493 seconds)
notificationworker stdout | 2024-04-22 03:38:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:22 UTC)" (scheduled at 2024-04-22 03:38:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:38:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:38:12,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 12, 431847), True, datetime.datetime(2024, 4, 22, 3, 38, 12, 431847), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:38:12,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:38:12,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:38:12,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:38:12,495 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:38:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:38:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:17.065745+00:00 (in 2.998173 seconds)
securityworker stdout | 2024-04-22 03:38:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:44 UTC)" (scheduled at 2024-04-22 03:38:14.067050+00:00)
securityworker stdout | 2024-04-22 03:38:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:38:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:38:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:38:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:38:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:38:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:38:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:38:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:38:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:38:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:38:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 33, 14, 76837), 1, 2])
securityworker stdout | 2024-04-22 03:38:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:38:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:38:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 33, 14, 76837), 1, 2])
securityworker stdout | 2024-04-22 03:38:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:38:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:38:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:38:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:38:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:38:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:38:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:38:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:38:14,941 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:38:15,251 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:38:15,630 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
gunicorn-web stdout | 2024-04-22 03:38:15,632 [285] [DEBUG] [app] Starting request: urn:request:30a9e2af-dc95-4c86-9f69-a56bafec2b19 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:15,632 [286] [DEBUG] [app] Starting request: urn:request:3026e321-438b-4b7e-803a-2e6f9afe61ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:38:15,636 [275] [DEBUG] [app] Starting request: urn:request:a2952696-4256-40b9-ae1c-48ddabc8839f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:15,637 [278] [DEBUG] [app] Starting request: urn:request:ca67970c-473b-47fc-817c-0e4e65dcae3f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:15,637 [275] [DEBUG] [app] Ending request: urn:request:a2952696-4256-40b9-ae1c-48ddabc8839f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2952696-4256-40b9-ae1c-48ddabc8839f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:15,637 [278] [DEBUG] [app] Ending request: urn:request:ca67970c-473b-47fc-817c-0e4e65dcae3f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ca67970c-473b-47fc-817c-0e4e65dcae3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:38:15,638 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:38:15,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:15,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:15,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:15,641 [284] [DEBUG] [app] Starting request: urn:request:e9d896e2-65cf-42c3-99ab-385af5c55395 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:15,642 [284] [DEBUG] [app] Ending request: urn:request:e9d896e2-65cf-42c3-99ab-385af5c55395 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9d896e2-65cf-42c3-99ab-385af5c55395', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:15,642 [282] [DEBUG] [app] Starting request: urn:request:add6af1b-d95d-46df-a36a-96cf95196404 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:15,642 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:38:15,642 [282] [DEBUG] [app] Ending request: urn:request:add6af1b-d95d-46df-a36a-96cf95196404 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:add6af1b-d95d-46df-a36a-96cf95196404', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:15,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:15,643 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:15,643 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.003)
gunicorn-web stdout | 2024-04-22 03:38:15,643 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:15,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:15,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:15,652 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:15,664 [286] [DEBUG] [app] Ending request: urn:request:3026e321-438b-4b7e-803a-2e6f9afe61ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3026e321-438b-4b7e-803a-2e6f9afe61ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:15,664 [285] [DEBUG] [app] Ending request: urn:request:30a9e2af-dc95-4c86-9f69-a56bafec2b19 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:30a9e2af-dc95-4c86-9f69-a56bafec2b19', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:15,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:38:15,665 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:38:15,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.033)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:38:15,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
exportactionlogsworker stdout | 2024-04-22 03:38:16,418 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:38:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:38:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:44.067050+00:00 (in 27.000728 seconds)
securityworker stdout | 2024-04-22 03:38:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:47 UTC)" (scheduled at 2024-04-22 03:38:17.065745+00:00)
securityworker stdout | 2024-04-22 03:38:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:38:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:38:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:38:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:38:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:38:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:38:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:38:17,475 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:38:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:38:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:53.045554+00:00 (in 34.997851 seconds)
exportactionlogsworker stdout | 2024-04-22 03:38:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:18 UTC)" (scheduled at 2024-04-22 03:38:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:38:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:38:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:38:18,351 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:38:18,847 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:38:19,334 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:38:19,338 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:38:19,344 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:38:19,949 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:38:20,381 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:38:20,513 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:38:20,768 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:38:20,824 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:38:20,831 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:38:20,841 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:38:20,848 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:38:20,854 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:38:20,856 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:38:20,859 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:38:20,864 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:38:21,043 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:38:21,049 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:38:21,052 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:38:21,060 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:38:21,065 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:38:21,246 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:38:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:32.430973+00:00 (in 9.999481 seconds)
notificationworker stdout | 2024-04-22 03:38:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:32 UTC)" (scheduled at 2024-04-22 03:38:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:38:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:38:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 22, 431825), True, datetime.datetime(2024, 4, 22, 3, 38, 22, 431825), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:38:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:38:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:38:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:38:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:38:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:24.738365+00:00 (in 59.999478 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:38:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:24 UTC)" (scheduled at 2024-04-22 03:38:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:38:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:38:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:38:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:38:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:38:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:02.035613+00:00 (in 35.999786 seconds)
repositorygcworker stdout | 2024-04-22 03:38:26,035 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:43:26 UTC)" (scheduled at 2024-04-22 03:38:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 03:38:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 38, 26, 36057), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:38:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 38, 26, 36057), True, datetime.datetime(2024, 4, 22, 3, 38, 26, 36057), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:38:26,054 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 38, 26, 36057), True, datetime.datetime(2024, 4, 22, 3, 38, 26, 36057), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 3, 38, 26, 36057), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:38:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:38:26,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:43:26 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:38:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:38:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:58.130973+00:00 (in 29.999407 seconds)
buildlogsarchiver stdout | 2024-04-22 03:38:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:58 UTC)" (scheduled at 2024-04-22 03:38:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:38:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 38, 28, 131952), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:38:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:38:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:38:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:38:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:30.933445+00:00 (in 1.001822 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:29 UTC)" (scheduled at 2024-04-22 03:38:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 29, 931860), True, datetime.datetime(2024, 4, 22, 3, 38, 29, 931860), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:38:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:38:30,632 [284] [DEBUG] [app] Starting request: urn:request:3ddd8c65-c055-4359-9c1d-e61bd7e0a331 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:30,632 [282] [DEBUG] [app] Starting request: urn:request:68892589-0784-449f-b399-2146b9cb7234 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:38:30,636 [283] [DEBUG] [app] Starting request: urn:request:90ed2321-93ce-44f5-a0ba-2778f4599334 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:30,636 [279] [DEBUG] [app] Starting request: urn:request:b70c0d2f-4fbc-4c3d-8327-e0fed6ec95ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:30,637 [279] [DEBUG] [app] Ending request: urn:request:b70c0d2f-4fbc-4c3d-8327-e0fed6ec95ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b70c0d2f-4fbc-4c3d-8327-e0fed6ec95ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:30,637 [283] [DEBUG] [app] Ending request: urn:request:90ed2321-93ce-44f5-a0ba-2778f4599334 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:90ed2321-93ce-44f5-a0ba-2778f4599334', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:30,637 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:38:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:38:30,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:30,639 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:30,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:30,640 [282] [DEBUG] [app] Starting request: urn:request:483d3c37-a06e-47b1-b875-4584c4e47149 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:30,641 [282] [DEBUG] [app] Ending request: urn:request:483d3c37-a06e-47b1-b875-4584c4e47149 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:483d3c37-a06e-47b1-b875-4584c4e47149', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:30,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:30,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:38:30,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:30,643 [286] [DEBUG] [app] Starting request: urn:request:76089e20-7f50-4e39-a037-6edb7cbfed7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:30,643 [286] [DEBUG] [app] Ending request: urn:request:76089e20-7f50-4e39-a037-6edb7cbfed7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76089e20-7f50-4e39-a037-6edb7cbfed7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:30,643 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:38:30,644 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:30,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:30,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:30,653 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:30,663 [284] [DEBUG] [app] Ending request: urn:request:3ddd8c65-c055-4359-9c1d-e61bd7e0a331 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3ddd8c65-c055-4359-9c1d-e61bd7e0a331', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:30,663 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.033)
gunicorn-web stdout | 2024-04-22 03:38:30,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:38:30,675 [282] [DEBUG] [app] Ending request: urn:request:68892589-0784-449f-b399-2146b9cb7234 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:68892589-0784-449f-b399-2146b9cb7234', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:30,675 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:38:30,675 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.044 118 0.043)
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:31.933721+00:00 (in 0.999804 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:43:30 UTC)" (scheduled at 2024-04-22 03:38:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 38, 30, 934149), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,948 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 38, 30, 934149), True, datetime.datetime(2024, 4, 22, 3, 38, 30, 934149), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,953 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 38, 30, 934149), True, datetime.datetime(2024, 4, 22, 3, 38, 30, 934149), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 3, 38, 30, 934149), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,957 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:38:30,957 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:43:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:38:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:38:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:01.842666+00:00 (in 29.999505 seconds)
gcworker stdout | 2024-04-22 03:38:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:01 UTC)" (scheduled at 2024-04-22 03:38:31.842666+00:00)
gcworker stdout | 2024-04-22 03:38:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:38:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:38:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:38:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:29.931002+00:00 (in 57.996793 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:38:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:31 UTC)" (scheduled at 2024-04-22 03:38:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:38:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:38:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:38:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:42.430973+00:00 (in 9.999555 seconds)
notificationworker stdout | 2024-04-22 03:38:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:42 UTC)" (scheduled at 2024-04-22 03:38:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:38:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:38:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 32, 431690), True, datetime.datetime(2024, 4, 22, 3, 38, 32, 431690), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:38:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:38:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:38:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:38:33,460 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:38:34,720 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:38:37,341 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:38:38,606 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:38:38,897 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:38:38,921 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:38:39,611 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:38:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:38:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:10.238173+00:00 (in 29.999458 seconds)
autopruneworker stdout | 2024-04-22 03:38:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:10 UTC)" (scheduled at 2024-04-22 03:38:40.238173+00:00)
teamsyncworker stdout | 2024-04-22 03:38:40,245 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
autopruneworker stdout | 2024-04-22 03:38:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757060247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:38:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:38:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:38:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:10 UTC)" executed successfully
storagereplication stdout | 2024-04-22 03:38:40,256 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:38:41,669 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:38:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:52.430973+00:00 (in 9.999468 seconds)
notificationworker stdout | 2024-04-22 03:38:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:52 UTC)" (scheduled at 2024-04-22 03:38:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:38:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:38:42,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 42, 431894), True, datetime.datetime(2024, 4, 22, 3, 38, 42, 431894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:38:42,453 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:38:42,453 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:38:42,453 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:38:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:38:42,533 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:38:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:38:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:47.065745+00:00 (in 2.998224 seconds)
securityworker stdout | 2024-04-22 03:38:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:14 UTC)" (scheduled at 2024-04-22 03:38:44.067050+00:00)
securityworker stdout | 2024-04-22 03:38:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:38:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:38:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:38:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:38:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:38:44,073 [95] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: quayregistry-clair-app.quay-enterprise.svc.cluster.local
securityworker stdout | 2024-04-22 03:38:44,085 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:38:44,086 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:38:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:38:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:38:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:38:44,105 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 33, 44, 86433), 1, 2])
securityworker stdout | 2024-04-22 03:38:44,108 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:38:44,108 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:38:44,109 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 33, 44, 86433), 1, 2])
securityworker stdout | 2024-04-22 03:38:44,113 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:38:44,113 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:38:44,113 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:44,113 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:38:44,113 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:38:44,113 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:38:44,113 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:38:44,115 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:38:44,115 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:38:44,115 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:38:44,978 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:38:45,288 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:38:45,632 [285] [DEBUG] [app] Starting request: urn:request:c07c96bd-844e-4f9b-b859-ecaf23de9191 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:45,632 [282] [DEBUG] [app] Starting request: urn:request:217b0a45-1c37-4f99-b822-55f0399f83ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:38:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:38:45,636 [275] [DEBUG] [app] Starting request: urn:request:a1dd9c3c-bccd-4074-bf09-2c9335c9a282 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:45,636 [281] [DEBUG] [app] Starting request: urn:request:557cc4ac-b39f-4a5d-9176-5b79c66f196c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:38:45,637 [281] [DEBUG] [app] Ending request: urn:request:557cc4ac-b39f-4a5d-9176-5b79c66f196c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:557cc4ac-b39f-4a5d-9176-5b79c66f196c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:45,637 [275] [DEBUG] [app] Ending request: urn:request:a1dd9c3c-bccd-4074-bf09-2c9335c9a282 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1dd9c3c-bccd-4074-bf09-2c9335c9a282', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:38:45,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:38:45,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:38:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:45,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:45,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:38:45,640 [285] [DEBUG] [app] Starting request: urn:request:300304bd-b465-4c04-ac9d-7b7309dd257e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:45,640 [285] [DEBUG] [app] Ending request: urn:request:300304bd-b465-4c04-ac9d-7b7309dd257e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:300304bd-b465-4c04-ac9d-7b7309dd257e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:45,640 [284] [DEBUG] [app] Starting request: urn:request:41062ce8-d8ab-4971-9440-31ad8da6ef76 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:38:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:38:45,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:38:45,641 [284] [DEBUG] [app] Ending request: urn:request:41062ce8-d8ab-4971-9440-31ad8da6ef76 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:41062ce8-d8ab-4971-9440-31ad8da6ef76', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:38:45,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:45,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:38:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:38:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:38:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:38:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:38:45,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:38:45,662 [285] [DEBUG] [app] Ending request: urn:request:c07c96bd-844e-4f9b-b859-ecaf23de9191 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c07c96bd-844e-4f9b-b859-ecaf23de9191', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:45,662 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:38:45,662 [282] [DEBUG] [app] Ending request: urn:request:217b0a45-1c37-4f99-b822-55f0399f83ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:217b0a45-1c37-4f99-b822-55f0399f83ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:38:45,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:38:45,662 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:38:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:38:45,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:38:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
globalpromstats stdout | 2024-04-22 03:38:45,666 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:38:46,455 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:38:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:38:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:14.067050+00:00 (in 27.000700 seconds)
securityworker stdout | 2024-04-22 03:38:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:17 UTC)" (scheduled at 2024-04-22 03:38:47.065745+00:00)
securityworker stdout | 2024-04-22 03:38:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:38:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:38:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:38:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:38:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:38:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:38:47,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:38:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:38:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:10.248668+00:00 (in 23.002240 seconds)
namespacegcworker stdout | 2024-04-22 03:38:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:47 UTC)" (scheduled at 2024-04-22 03:38:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:38:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:38:47,248 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 47, 246742), True, datetime.datetime(2024, 4, 22, 3, 38, 47, 246742), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:38:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:38:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:38:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:38:47,487 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:38:48,388 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:38:48,882 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:38:49,345 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:38:49,349 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:38:49,353 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:38:49,988 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:38:50,418 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:38:50,553 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:38:50,782 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:38:50,838 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:38:50,846 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:38:50,855 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:38:50,860 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:38:50,865 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:38:50,869 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:38:50,873 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:38:50,881 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:38:51,058 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:38:51,063 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:38:51,067 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:38:51,070 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:38:51,074 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:38:51,284 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:38:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:38:54.433683+00:00 (in 2.002175 seconds)
notificationworker stdout | 2024-04-22 03:38:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:02 UTC)" (scheduled at 2024-04-22 03:38:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:38:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:38:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 52, 431757), True, datetime.datetime(2024, 4, 22, 3, 38, 52, 431757), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:38:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:38:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:38:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:38:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:38:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:18.047111+00:00 (in 25.001047 seconds)
exportactionlogsworker stdout | 2024-04-22 03:38:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:53 UTC)" (scheduled at 2024-04-22 03:38:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:38:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:38:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 38, 53, 46527), True, datetime.datetime(2024, 4, 22, 3, 38, 53, 46527), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:38:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:38:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:38:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:38:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:38:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:02.430973+00:00 (in 7.996664 seconds)
notificationworker stdout | 2024-04-22 03:38:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:54 UTC)" (scheduled at 2024-04-22 03:38:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:38:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:38:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:39:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:38:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:38:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:28.130973+00:00 (in 29.999378 seconds)
buildlogsarchiver stdout | 2024-04-22 03:38:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:28 UTC)" (scheduled at 2024-04-22 03:38:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:38:58,133 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 38, 58, 132132), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:38:58,148 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:38:58,148 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:38:58,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:39:00,632 [286] [DEBUG] [app] Starting request: urn:request:78b7f2e7-7298-41f8-8853-9515c85d25ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:00,632 [285] [DEBUG] [app] Starting request: urn:request:0cf9b9bc-409d-46d4-ab62-e693f55007e9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:39:00,636 [283] [DEBUG] [app] Starting request: urn:request:fe7c9c0c-9f18-48ae-8876-f6bfae40644d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:00,636 [275] [DEBUG] [app] Starting request: urn:request:aba47239-7295-46eb-b5bc-1bfb4ec9a683 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:00,636 [275] [DEBUG] [app] Ending request: urn:request:aba47239-7295-46eb-b5bc-1bfb4ec9a683 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aba47239-7295-46eb-b5bc-1bfb4ec9a683', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:00,636 [283] [DEBUG] [app] Ending request: urn:request:fe7c9c0c-9f18-48ae-8876-f6bfae40644d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fe7c9c0c-9f18-48ae-8876-f6bfae40644d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:39:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:00,639 [286] [DEBUG] [app] Starting request: urn:request:727fd10b-7c1d-4a59-8973-6a908994d021 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:00,639 [286] [DEBUG] [app] Ending request: urn:request:727fd10b-7c1d-4a59-8973-6a908994d021 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:727fd10b-7c1d-4a59-8973-6a908994d021', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:39:00,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:00,640 [282] [DEBUG] [app] Starting request: urn:request:a43ec8a5-f13e-4544-8d46-fbca17fba59e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:00,640 [282] [DEBUG] [app] Ending request: urn:request:a43ec8a5-f13e-4544-8d46-fbca17fba59e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a43ec8a5-f13e-4544-8d46-fbca17fba59e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:39:00,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:00,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:00,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:00,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:00,661 [285] [DEBUG] [app] Ending request: urn:request:0cf9b9bc-409d-46d4-ab62-e693f55007e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0cf9b9bc-409d-46d4-ab62-e693f55007e9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:00,662 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:39:00,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:39:00,671 [286] [DEBUG] [app] Ending request: urn:request:78b7f2e7-7298-41f8-8853-9515c85d25ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78b7f2e7-7298-41f8-8853-9515c85d25ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:00,672 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:00,672 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.041 118 0.041)
gcworker stdout | 2024-04-22 03:39:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:39:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:31.842666+00:00 (in 29.999504 seconds)
gcworker stdout | 2024-04-22 03:39:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:31 UTC)" (scheduled at 2024-04-22 03:39:01.842666+00:00)
gcworker stdout | 2024-04-22 03:39:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:39:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:39:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:39:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:09.033206+00:00 (in 6.997054 seconds)
repositorygcworker stdout | 2024-04-22 03:39:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:02 UTC)" (scheduled at 2024-04-22 03:39:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:39:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:39:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:39:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:12.430973+00:00 (in 9.999487 seconds)
notificationworker stdout | 2024-04-22 03:39:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:12 UTC)" (scheduled at 2024-04-22 03:39:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:39:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:39:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 2, 431793), True, datetime.datetime(2024, 4, 22, 3, 39, 2, 431793), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:39:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:39:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:39:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:39:03,467 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:39:04,759 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:39:07,377 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:39:08,642 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:39:08,934 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:39:08,957 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:39:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:39:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:02.035613+00:00 (in 53.001909 seconds)
repositorygcworker stdout | 2024-04-22 03:39:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:09 UTC)" (scheduled at 2024-04-22 03:39:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:39:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:39:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 9, 33966), True, datetime.datetime(2024, 4, 22, 3, 39, 9, 33966), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:39:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:39:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:39:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:39:09,645 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:39:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:39:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:40.238173+00:00 (in 29.999501 seconds)
autopruneworker stdout | 2024-04-22 03:39:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:40 UTC)" (scheduled at 2024-04-22 03:39:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:39:10,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757090247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:39:10,249 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:39:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:47.245913+00:00 (in 36.996547 seconds)
namespacegcworker stdout | 2024-04-22 03:39:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:10 UTC)" (scheduled at 2024-04-22 03:39:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:39:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:39:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:39:10,255 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:39:10,255 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:39:10,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:39:10,284 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:39:10,295 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:39:11,705 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:39:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:22.430973+00:00 (in 9.999478 seconds)
notificationworker stdout | 2024-04-22 03:39:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:22 UTC)" (scheduled at 2024-04-22 03:39:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:39:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:39:12,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 12, 431880), True, datetime.datetime(2024, 4, 22, 3, 39, 12, 431880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:39:12,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:39:12,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:39:12,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:39:12,569 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:39:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:39:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:17.065745+00:00 (in 2.998170 seconds)
securityworker stdout | 2024-04-22 03:39:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:44 UTC)" (scheduled at 2024-04-22 03:39:14.067050+00:00)
securityworker stdout | 2024-04-22 03:39:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:39:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:39:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:39:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:39:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:39:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:39:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:39:14,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:39:14,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:39:14,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:39:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 34, 14, 78471), 1, 2])
securityworker stdout | 2024-04-22 03:39:14,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:39:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:39:14,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 34, 14, 78471), 1, 2])
securityworker stdout | 2024-04-22 03:39:14,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:39:14,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:39:14,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:14,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:39:14,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:39:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:39:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:39:14,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:14,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:39:14,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:39:15,018 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:39:15,325 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:39:15,632 [282] [DEBUG] [app] Starting request: urn:request:4887680c-965c-4a00-b1ab-3c406a666dd0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:15,632 [285] [DEBUG] [app] Starting request: urn:request:7f85539e-d959-45d6-8924-1c135d65f2c6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:39:15,635 [275] [DEBUG] [app] Starting request: urn:request:b2df45f3-ea87-4d4d-a97c-957cefad282b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:15,635 [277] [DEBUG] [app] Starting request: urn:request:a0046f7a-1cb6-4eb0-83a3-1e5f14caeb4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:15,636 [277] [DEBUG] [app] Ending request: urn:request:a0046f7a-1cb6-4eb0-83a3-1e5f14caeb4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0046f7a-1cb6-4eb0-83a3-1e5f14caeb4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:15,636 [275] [DEBUG] [app] Ending request: urn:request:b2df45f3-ea87-4d4d-a97c-957cefad282b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2df45f3-ea87-4d4d-a97c-957cefad282b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:39:15,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:39:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:15,639 [284] [DEBUG] [app] Starting request: urn:request:76e1fba2-6771-4f61-8aa0-bb2518a846c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:15,639 [286] [DEBUG] [app] Starting request: urn:request:61c31a76-37c9-4e12-a953-5c95f910aadc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:15,640 [284] [DEBUG] [app] Ending request: urn:request:76e1fba2-6771-4f61-8aa0-bb2518a846c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76e1fba2-6771-4f61-8aa0-bb2518a846c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:15,640 [286] [DEBUG] [app] Ending request: urn:request:61c31a76-37c9-4e12-a953-5c95f910aadc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:61c31a76-37c9-4e12-a953-5c95f910aadc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:39:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:15,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:15,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:15,661 [282] [DEBUG] [app] Ending request: urn:request:4887680c-965c-4a00-b1ab-3c406a666dd0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4887680c-965c-4a00-b1ab-3c406a666dd0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:15,662 [285] [DEBUG] [app] Ending request: urn:request:7f85539e-d959-45d6-8924-1c135d65f2c6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f85539e-d959-45d6-8924-1c135d65f2c6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:15,662 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:15,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:39:15,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
globalpromstats stdout | 2024-04-22 03:39:15,697 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:39:16,463 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:39:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:39:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:44.067050+00:00 (in 27.000827 seconds)
securityworker stdout | 2024-04-22 03:39:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:47 UTC)" (scheduled at 2024-04-22 03:39:17.065745+00:00)
securityworker stdout | 2024-04-22 03:39:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:39:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:39:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:39:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:39:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:39:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:39:17,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:39:17,523 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:39:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:39:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:53.045554+00:00 (in 34.997909 seconds)
exportactionlogsworker stdout | 2024-04-22 03:39:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:18 UTC)" (scheduled at 2024-04-22 03:39:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:39:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:39:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:39:18,396 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:39:18,895 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:39:19,354 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:39:19,357 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:39:19,360 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:39:20,024 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:39:20,442 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:39:20,593 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:39:20,793 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:39:20,852 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:39:20,859 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:39:20,870 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:39:20,877 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:39:20,884 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:39:20,888 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:39:20,893 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:39:20,895 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:39:21,067 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:39:21,072 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:39:21,076 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:39:21,081 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:39:21,085 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:39:21,304 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:39:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:32.430973+00:00 (in 9.999384 seconds)
notificationworker stdout | 2024-04-22 03:39:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:32 UTC)" (scheduled at 2024-04-22 03:39:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:39:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:39:22,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 22, 431929), True, datetime.datetime(2024, 4, 22, 3, 39, 22, 431929), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:39:22,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:39:22,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:39:22,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:39:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:39:24,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:24.738365+00:00 (in 59.999360 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:39:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:24 UTC)" (scheduled at 2024-04-22 03:39:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:39:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:39:24,753 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:39:24,753 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:39:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:39:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:58.130973+00:00 (in 29.999437 seconds)
buildlogsarchiver stdout | 2024-04-22 03:39:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:58 UTC)" (scheduled at 2024-04-22 03:39:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:39:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 39, 28, 131809), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:39:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:39:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:39:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:39:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:31.933721+00:00 (in 2.002176 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:29 UTC)" (scheduled at 2024-04-22 03:39:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 29, 931827), True, datetime.datetime(2024, 4, 22, 3, 39, 29, 931827), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:39:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:39:30,632 [284] [DEBUG] [app] Starting request: urn:request:9b52af72-1da7-485c-ab03-36bc5c1cd0e1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:30,632 [282] [DEBUG] [app] Starting request: urn:request:cd0da8c9-861c-4eca-a879-e6cfa2bc2e70 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:39:30,635 [283] [DEBUG] [app] Starting request: urn:request:dd52cf1c-ac6d-4dd1-9d2a-cb6e37bae1ac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:30,636 [283] [DEBUG] [app] Ending request: urn:request:dd52cf1c-ac6d-4dd1-9d2a-cb6e37bae1ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dd52cf1c-ac6d-4dd1-9d2a-cb6e37bae1ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:30,636 [275] [DEBUG] [app] Starting request: urn:request:bc1a71eb-ba79-4051-bd19-a6a604912652 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:39:30,636 [275] [DEBUG] [app] Ending request: urn:request:bc1a71eb-ba79-4051-bd19-a6a604912652 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bc1a71eb-ba79-4051-bd19-a6a604912652', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:39:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:39:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:39:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:30,639 [286] [DEBUG] [app] Starting request: urn:request:8daf2295-9e3c-4096-b6bf-e4e70e9f870a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:30,640 [286] [DEBUG] [app] Ending request: urn:request:8daf2295-9e3c-4096-b6bf-e4e70e9f870a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8daf2295-9e3c-4096-b6bf-e4e70e9f870a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:30,640 [285] [DEBUG] [app] Starting request: urn:request:d25b8c89-5e20-449a-92f5-0573002b6bf3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:39:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:30,640 [285] [DEBUG] [app] Ending request: urn:request:d25b8c89-5e20-449a-92f5-0573002b6bf3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d25b8c89-5e20-449a-92f5-0573002b6bf3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:30,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:39:30,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:30,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:30,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:30,663 [282] [DEBUG] [app] Ending request: urn:request:cd0da8c9-861c-4eca-a879-e6cfa2bc2e70 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd0da8c9-861c-4eca-a879-e6cfa2bc2e70', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:30,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:30,663 [284] [DEBUG] [app] Ending request: urn:request:9b52af72-1da7-485c-ab03-36bc5c1cd0e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9b52af72-1da7-485c-ab03-36bc5c1cd0e1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:30,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:30,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:39:30,663 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gcworker stdout | 2024-04-22 03:39:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:39:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:01.842666+00:00 (in 29.999374 seconds)
gcworker stdout | 2024-04-22 03:39:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:01 UTC)" (scheduled at 2024-04-22 03:39:31.842666+00:00)
gcworker stdout | 2024-04-22 03:39:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:39:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:39:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:39:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:29.931002+00:00 (in 57.996569 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:39:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:31 UTC)" (scheduled at 2024-04-22 03:39:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:39:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:39:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:39:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:42.430973+00:00 (in 9.999330 seconds)
notificationworker stdout | 2024-04-22 03:39:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:42 UTC)" (scheduled at 2024-04-22 03:39:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:39:32,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:39:32,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 32, 432223), True, datetime.datetime(2024, 4, 22, 3, 39, 32, 432223), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:39:32,457 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:39:32,457 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:39:32,457 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:39:33,503 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:39:34,801 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:39:37,402 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:39:38,649 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:39:38,967 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:39:38,993 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:39:39,658 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:39:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:39:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:10.238173+00:00 (in 29.999517 seconds)
autopruneworker stdout | 2024-04-22 03:39:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:10 UTC)" (scheduled at 2024-04-22 03:39:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:39:40,261 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757120261, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:39:40,267 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:39:40,267 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:39:40,267 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:39:40,297 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:39:40,324 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:39:41,742 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:39:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:52.430973+00:00 (in 9.999422 seconds)
notificationworker stdout | 2024-04-22 03:39:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:52 UTC)" (scheduled at 2024-04-22 03:39:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:39:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:39:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 42, 431799), True, datetime.datetime(2024, 4, 22, 3, 39, 42, 431799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:39:42,456 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:39:42,456 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:39:42,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:39:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:39:42,609 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:39:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:39:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:47.065745+00:00 (in 2.998206 seconds)
securityworker stdout | 2024-04-22 03:39:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:14 UTC)" (scheduled at 2024-04-22 03:39:44.067050+00:00)
securityworker stdout | 2024-04-22 03:39:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:39:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:39:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:39:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:39:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:39:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:39:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:39:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:39:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:39:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:39:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 34, 44, 77449), 1, 2])
securityworker stdout | 2024-04-22 03:39:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:39:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:39:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 34, 44, 77449), 1, 2])
securityworker stdout | 2024-04-22 03:39:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:39:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:39:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:39:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:39:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:39:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:39:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:39:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:39:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:39:45,048 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:39:45,345 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:39:45,632 [282] [DEBUG] [app] Starting request: urn:request:25dae8a1-c871-484f-8740-f9fd60a8236f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:45,632 [284] [DEBUG] [app] Starting request: urn:request:8b58eca4-a674-4dc3-a2ac-0d984d7dbc66 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:39:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:39:45,637 [275] [DEBUG] [app] Starting request: urn:request:d2efd9ae-9110-403c-ad8d-e8c66eb7b4eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:45,637 [281] [DEBUG] [app] Starting request: urn:request:abf02975-8375-4ce7-8005-deb0902c3ba9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:39:45,637 [275] [DEBUG] [app] Ending request: urn:request:d2efd9ae-9110-403c-ad8d-e8c66eb7b4eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2efd9ae-9110-403c-ad8d-e8c66eb7b4eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:45,637 [281] [DEBUG] [app] Ending request: urn:request:abf02975-8375-4ce7-8005-deb0902c3ba9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abf02975-8375-4ce7-8005-deb0902c3ba9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:39:45,638 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:39:45,638 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:39:45,638 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:45,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:45,640 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:45,640 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:39:45,642 [284] [DEBUG] [app] Starting request: urn:request:ae211365-fc0f-4e30-b626-303e31fe15f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:45,642 [286] [DEBUG] [app] Starting request: urn:request:072db993-943a-410c-8924-8741e18f5399 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:39:45,643 [284] [DEBUG] [app] Ending request: urn:request:ae211365-fc0f-4e30-b626-303e31fe15f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae211365-fc0f-4e30-b626-303e31fe15f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:45,643 [286] [DEBUG] [app] Ending request: urn:request:072db993-943a-410c-8924-8741e18f5399 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:072db993-943a-410c-8924-8741e18f5399', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:39:45,643 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:39:45,643 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:39:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:39:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:39:45,644 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:45,644 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:39:45,644 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:45,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:39:45,654 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:45,654 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:39:45,666 [282] [DEBUG] [app] Ending request: urn:request:25dae8a1-c871-484f-8740-f9fd60a8236f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:25dae8a1-c871-484f-8740-f9fd60a8236f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:45,666 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:45,666 [284] [DEBUG] [app] Ending request: urn:request:8b58eca4-a674-4dc3-a2ac-0d984d7dbc66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8b58eca4-a674-4dc3-a2ac-0d984d7dbc66', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:39:45,666 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:39:45,666 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.035)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:39:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.036)
gunicorn-web stdout | 2024-04-22 03:39:45,667 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:39:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
globalpromstats stdout | 2024-04-22 03:39:45,737 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:39:46,503 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:39:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:39:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:14.067050+00:00 (in 27.000693 seconds)
securityworker stdout | 2024-04-22 03:39:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:17 UTC)" (scheduled at 2024-04-22 03:39:47.065745+00:00)
securityworker stdout | 2024-04-22 03:39:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:39:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:39:47,070 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:39:47,072 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:39:47,085 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:39:47,088 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:39:47,089 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:39:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:39:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:10.248668+00:00 (in 23.002123 seconds)
namespacegcworker stdout | 2024-04-22 03:39:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:47 UTC)" (scheduled at 2024-04-22 03:39:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:39:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:39:47,248 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 47, 246994), True, datetime.datetime(2024, 4, 22, 3, 39, 47, 246994), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:39:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:39:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:39:47,263 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:39:47,560 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:39:48,432 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:39:48,931 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:39:49,365 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:39:49,369 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:39:49,372 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:39:50,062 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:39:50,478 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:39:50,606 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:39:50,802 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:39:50,866 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:39:50,871 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:39:50,880 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:39:50,891 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:39:50,894 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:39:50,898 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:39:50,903 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:39:50,906 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:39:51,076 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:39:51,079 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:39:51,083 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:39:51,089 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:39:51,092 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:39:51,313 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:39:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:39:54.433683+00:00 (in 2.002151 seconds)
notificationworker stdout | 2024-04-22 03:39:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:02 UTC)" (scheduled at 2024-04-22 03:39:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:39:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:39:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 52, 431901), True, datetime.datetime(2024, 4, 22, 3, 39, 52, 431901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:39:52,455 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:39:52,455 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:39:52,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:39:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:39:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:18.047111+00:00 (in 25.001044 seconds)
exportactionlogsworker stdout | 2024-04-22 03:39:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:53 UTC)" (scheduled at 2024-04-22 03:39:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:39:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:39:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 39, 53, 46389), True, datetime.datetime(2024, 4, 22, 3, 39, 53, 46389), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:39:53,069 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:39:53,069 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:39:53,069 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:39:54,434 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:39:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:02.430973+00:00 (in 7.996599 seconds)
notificationworker stdout | 2024-04-22 03:39:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:54 UTC)" (scheduled at 2024-04-22 03:39:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:39:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:39:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:40:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:39:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:39:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:28.130973+00:00 (in 29.999495 seconds)
buildlogsarchiver stdout | 2024-04-22 03:39:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:28 UTC)" (scheduled at 2024-04-22 03:39:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:39:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 39, 58, 131736), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:39:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:39:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:39:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:40:00,632 [282] [DEBUG] [app] Starting request: urn:request:7dd3a3a0-b5fa-4320-8829-d8d94b2911ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:00,632 [286] [DEBUG] [app] Starting request: urn:request:5b3c6648-9d7c-4b6e-b0d6-d009d2c91026 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:40:00,637 [278] [DEBUG] [app] Starting request: urn:request:e387d307-c806-4c9d-bdd7-aadf9638c858 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:00,637 [278] [DEBUG] [app] Ending request: urn:request:e387d307-c806-4c9d-bdd7-aadf9638c858 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e387d307-c806-4c9d-bdd7-aadf9638c858', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:40:00,638 [275] [DEBUG] [app] Starting request: urn:request:517f8422-11d8-4b04-9bc1-e41678b09636 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:00,638 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:40:00,638 [275] [DEBUG] [app] Ending request: urn:request:517f8422-11d8-4b04-9bc1-e41678b09636 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:517f8422-11d8-4b04-9bc1-e41678b09636', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:00,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.002)
gunicorn-web stdout | 2024-04-22 03:40:00,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:40:00,640 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.003)
gunicorn-web stdout | 2024-04-22 03:40:00,640 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:00,641 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:00,642 [285] [DEBUG] [app] Starting request: urn:request:e0601dcc-47ab-4d2b-8c08-3ef557d3de21 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:00,643 [285] [DEBUG] [app] Ending request: urn:request:e0601dcc-47ab-4d2b-8c08-3ef557d3de21 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0601dcc-47ab-4d2b-8c08-3ef557d3de21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:00,643 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.002)
gunicorn-web stdout | 2024-04-22 03:40:00,644 [282] [DEBUG] [app] Starting request: urn:request:a1d2ae99-1c91-4407-b5ab-dac79fca840b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:00,644 [282] [DEBUG] [app] Ending request: urn:request:a1d2ae99-1c91-4407-b5ab-dac79fca840b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1d2ae99-1c91-4407-b5ab-dac79fca840b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:00,645 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.002)
gunicorn-web stdout | 2024-04-22 03:40:00,645 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:00,645 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:00,646 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:00,646 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:00,655 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:00,655 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:00,669 [286] [DEBUG] [app] Ending request: urn:request:5b3c6648-9d7c-4b6e-b0d6-d009d2c91026 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b3c6648-9d7c-4b6e-b0d6-d009d2c91026', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:00,669 [282] [DEBUG] [app] Ending request: urn:request:7dd3a3a0-b5fa-4320-8829-d8d94b2911ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7dd3a3a0-b5fa-4320-8829-d8d94b2911ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:00,669 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:40:00,669 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.039)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.039)
gunicorn-web stdout | 2024-04-22 03:40:00,670 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:40:00,670 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:40:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:40:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:31.842666+00:00 (in 29.999386 seconds)
gcworker stdout | 2024-04-22 03:40:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:31 UTC)" (scheduled at 2024-04-22 03:40:01.842666+00:00)
gcworker stdout | 2024-04-22 03:40:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:40:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:40:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:40:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:09.033206+00:00 (in 6.997100 seconds)
repositorygcworker stdout | 2024-04-22 03:40:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:02 UTC)" (scheduled at 2024-04-22 03:40:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:40:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:40:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:40:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:12.430973+00:00 (in 9.999118 seconds)
notificationworker stdout | 2024-04-22 03:40:02,432 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:12 UTC)" (scheduled at 2024-04-22 03:40:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:40:02,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:40:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 2, 432365), True, datetime.datetime(2024, 4, 22, 3, 40, 2, 432365), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:40:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:40:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:40:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:40:03,540 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:40:04,808 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:40:07,436 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:40:08,685 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:40:08,979 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:40:09,022 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
repositorygcworker stdout | 2024-04-22 03:40:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:40:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:02.035613+00:00 (in 53.001921 seconds)
repositorygcworker stdout | 2024-04-22 03:40:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:09 UTC)" (scheduled at 2024-04-22 03:40:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:40:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:40:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 9, 33942), True, datetime.datetime(2024, 4, 22, 3, 40, 9, 33942), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:40:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:40:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:40:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:09 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:40:09,697 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:40:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:40:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:40.238173+00:00 (in 29.999461 seconds)
autopruneworker stdout | 2024-04-22 03:40:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:40 UTC)" (scheduled at 2024-04-22 03:40:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:40:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757150248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:40:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:40:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:47.245913+00:00 (in 36.996714 seconds)
namespacegcworker stdout | 2024-04-22 03:40:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:10 UTC)" (scheduled at 2024-04-22 03:40:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:40:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:40:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:40:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:40:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:40:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:40:10,335 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:40:10,360 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:40:11,780 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:40:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:22.430973+00:00 (in 9.999318 seconds)
notificationworker stdout | 2024-04-22 03:40:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:22 UTC)" (scheduled at 2024-04-22 03:40:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:40:12,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:40:12,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 12, 432363), True, datetime.datetime(2024, 4, 22, 3, 40, 12, 432363), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:40:12,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:40:12,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:40:12,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:40:12,642 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:40:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:40:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:17.065745+00:00 (in 2.998096 seconds)
securityworker stdout | 2024-04-22 03:40:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:44 UTC)" (scheduled at 2024-04-22 03:40:14.067050+00:00)
securityworker stdout | 2024-04-22 03:40:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:40:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:40:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:40:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:40:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:40:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:40:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:40:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:40:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:40:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:40:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 35, 14, 77947), 1, 2])
securityworker stdout | 2024-04-22 03:40:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:40:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:40:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 35, 14, 77947), 1, 2])
securityworker stdout | 2024-04-22 03:40:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:40:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:40:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:40:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:40:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:40:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:40:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:40:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:40:15,057 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:40:15,382 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:40:15,632 [286] [DEBUG] [app] Starting request: urn:request:66d7f05f-46f7-45e6-bd72-eac5904f4bd1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:15,632 [284] [DEBUG] [app] Starting request: urn:request:0b66b2fb-b303-4263-b806-c1d97d096fc7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:40:15,635 [275] [DEBUG] [app] Starting request: urn:request:4dfec5b1-d3ff-4d19-a60b-4552ff671731 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:15,635 [281] [DEBUG] [app] Starting request: urn:request:5d13fe35-0cb8-4220-a8de-edabad72decd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:15,636 [281] [DEBUG] [app] Ending request: urn:request:5d13fe35-0cb8-4220-a8de-edabad72decd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d13fe35-0cb8-4220-a8de-edabad72decd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:40:15,636 [275] [DEBUG] [app] Ending request: urn:request:4dfec5b1-d3ff-4d19-a60b-4552ff671731 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4dfec5b1-d3ff-4d19-a60b-4552ff671731', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:40:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:40:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:15,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:15,639 [286] [DEBUG] [app] Starting request: urn:request:bbd8fc2e-d158-420b-b935-84ea45f389ec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:15,639 [285] [DEBUG] [app] Starting request: urn:request:ff41b1c8-c2af-452a-ab94-eb55b0de5459 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:15,639 [286] [DEBUG] [app] Ending request: urn:request:bbd8fc2e-d158-420b-b935-84ea45f389ec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bbd8fc2e-d158-420b-b935-84ea45f389ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:15,639 [285] [DEBUG] [app] Ending request: urn:request:ff41b1c8-c2af-452a-ab94-eb55b0de5459 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff41b1c8-c2af-452a-ab94-eb55b0de5459', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:40:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:15,663 [284] [DEBUG] [app] Ending request: urn:request:0b66b2fb-b303-4263-b806-c1d97d096fc7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b66b2fb-b303-4263-b806-c1d97d096fc7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:15,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:40:15,664 [286] [DEBUG] [app] Ending request: urn:request:66d7f05f-46f7-45e6-bd72-eac5904f4bd1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:66d7f05f-46f7-45e6-bd72-eac5904f4bd1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:15,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:40:15,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:40:15,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
globalpromstats stdout | 2024-04-22 03:40:15,773 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:40:16,536 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:40:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:40:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:44.067050+00:00 (in 27.000721 seconds)
securityworker stdout | 2024-04-22 03:40:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:47 UTC)" (scheduled at 2024-04-22 03:40:17.065745+00:00)
securityworker stdout | 2024-04-22 03:40:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:40:17,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:40:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:40:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:40:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:40:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:40:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:40:17,567 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:40:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:40:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:53.045554+00:00 (in 34.997735 seconds)
exportactionlogsworker stdout | 2024-04-22 03:40:18,048 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:18 UTC)" (scheduled at 2024-04-22 03:40:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:40:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:40:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:40:18,445 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:40:18,958 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:40:19,374 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:40:19,377 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:40:19,380 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:40:20,096 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:40:20,517 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:40:20,647 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:40:20,815 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:40:20,876 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:40:20,883 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:40:20,892 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:40:20,905 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:40:20,911 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:40:20,916 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:40:20,921 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:40:20,926 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:40:21,085 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:40:21,089 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:40:21,093 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:40:21,100 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:40:21,103 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:40:21,353 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:40:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:32.430973+00:00 (in 9.999412 seconds)
notificationworker stdout | 2024-04-22 03:40:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:32 UTC)" (scheduled at 2024-04-22 03:40:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:40:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:40:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 22, 431929), True, datetime.datetime(2024, 4, 22, 3, 40, 22, 431929), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:40:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:40:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:40:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:40:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:40:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:24.738365+00:00 (in 59.999397 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:40:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:24 UTC)" (scheduled at 2024-04-22 03:40:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:40:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:40:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:40:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:40:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:40:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:58.130973+00:00 (in 29.999524 seconds)
buildlogsarchiver stdout | 2024-04-22 03:40:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:58 UTC)" (scheduled at 2024-04-22 03:40:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:40:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 40, 28, 131672), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:40:28,149 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:40:28,149 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:40:28,149 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:40:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:31.933721+00:00 (in 2.002134 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:29 UTC)" (scheduled at 2024-04-22 03:40:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 29, 931918), True, datetime.datetime(2024, 4, 22, 3, 40, 29, 931918), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:40:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:40:30,632 [286] [DEBUG] [app] Starting request: urn:request:3fd830bb-e3d6-4718-94b8-c5c09e7e0e79 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:30,632 [282] [DEBUG] [app] Starting request: urn:request:d12bd06d-d07f-4d28-b9df-082bdaf42619 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:40:30,635 [280] [DEBUG] [app] Starting request: urn:request:adb264b6-c1d4-49dc-abf2-cab2cf6c3f17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:30,635 [279] [DEBUG] [app] Starting request: urn:request:0f06fef8-ae00-4027-8aca-4d032184c908 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:30,636 [280] [DEBUG] [app] Ending request: urn:request:adb264b6-c1d4-49dc-abf2-cab2cf6c3f17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:adb264b6-c1d4-49dc-abf2-cab2cf6c3f17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:40:30,636 [279] [DEBUG] [app] Ending request: urn:request:0f06fef8-ae00-4027-8aca-4d032184c908 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f06fef8-ae00-4027-8aca-4d032184c908', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:40:30,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:40:30,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:40:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:30,639 [285] [DEBUG] [app] Starting request: urn:request:e6589fdc-4d0b-4727-8d7b-a084fb04fc65 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:30,639 [284] [DEBUG] [app] Starting request: urn:request:82f8dea1-0435-44c8-b2f1-214861af47c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:30,639 [285] [DEBUG] [app] Ending request: urn:request:e6589fdc-4d0b-4727-8d7b-a084fb04fc65 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e6589fdc-4d0b-4727-8d7b-a084fb04fc65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:30,639 [284] [DEBUG] [app] Ending request: urn:request:82f8dea1-0435-44c8-b2f1-214861af47c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:82f8dea1-0435-44c8-b2f1-214861af47c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:40:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:30,661 [286] [DEBUG] [app] Ending request: urn:request:3fd830bb-e3d6-4718-94b8-c5c09e7e0e79 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3fd830bb-e3d6-4718-94b8-c5c09e7e0e79', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:30,661 [282] [DEBUG] [app] Ending request: urn:request:d12bd06d-d07f-4d28-b9df-082bdaf42619 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d12bd06d-d07f-4d28-b9df-082bdaf42619', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:40:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:40:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:40:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 03:40:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:40:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:01.842666+00:00 (in 29.999447 seconds)
gcworker stdout | 2024-04-22 03:40:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:01 UTC)" (scheduled at 2024-04-22 03:40:31.842666+00:00)
gcworker stdout | 2024-04-22 03:40:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:40:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:40:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:40:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:29.931002+00:00 (in 57.996801 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:40:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:31 UTC)" (scheduled at 2024-04-22 03:40:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:40:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:40:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:40:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:42.430973+00:00 (in 9.999515 seconds)
notificationworker stdout | 2024-04-22 03:40:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:42 UTC)" (scheduled at 2024-04-22 03:40:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:40:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:40:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 32, 431897), True, datetime.datetime(2024, 4, 22, 3, 40, 32, 431897), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:40:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:40:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:40:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:40:33,580 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:40:34,844 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:40:37,472 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:40:38,722 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:40:39,016 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:40:39,058 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:40:39,715 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:40:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:40:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:10.238173+00:00 (in 29.999481 seconds)
autopruneworker stdout | 2024-04-22 03:40:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:10 UTC)" (scheduled at 2024-04-22 03:40:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:40:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757180247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:40:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:40:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:40:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:40:40,342 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:40:40,401 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:40:41,808 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:40:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:52.430973+00:00 (in 9.999420 seconds)
notificationworker stdout | 2024-04-22 03:40:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:52 UTC)" (scheduled at 2024-04-22 03:40:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:40:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:40:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 42, 431947), True, datetime.datetime(2024, 4, 22, 3, 40, 42, 431947), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:40:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:40:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:40:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:40:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:40:42,679 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:40:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:40:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:47.065745+00:00 (in 2.998152 seconds)
securityworker stdout | 2024-04-22 03:40:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:14 UTC)" (scheduled at 2024-04-22 03:40:44.067050+00:00)
securityworker stdout | 2024-04-22 03:40:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:40:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:40:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:40:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:40:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:40:44,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:40:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:40:44,093 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:40:44,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:40:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:40:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 35, 44, 78569), 1, 2])
securityworker stdout | 2024-04-22 03:40:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:40:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:40:44,104 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 35, 44, 78569), 1, 2])
securityworker stdout | 2024-04-22 03:40:44,108 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:40:44,108 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:40:44,108 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:44,108 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:40:44,109 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:40:44,109 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:40:44,109 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:40:44,110 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:40:44,110 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:40:44,110 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:40:45,097 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:40:45,418 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:40:45,632 [284] [DEBUG] [app] Starting request: urn:request:9df322a3-9eb1-4139-9674-70b8ac81df46 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:45,634 [286] [DEBUG] [app] Starting request: urn:request:3b2f0c66-409f-40b3-8804-3f852665f75d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:40:45,635 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:45,636 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:40:45,637 [280] [DEBUG] [app] Starting request: urn:request:5dfe71aa-9828-4387-8d7f-dc7a4f6d320a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:40:45,638 [280] [DEBUG] [app] Ending request: urn:request:5dfe71aa-9828-4387-8d7f-dc7a4f6d320a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5dfe71aa-9828-4387-8d7f-dc7a4f6d320a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:45,638 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:40:45,639 [275] [DEBUG] [app] Starting request: urn:request:5c96540d-2700-4818-8b63-9ad06ced1f8c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.003)
gunicorn-registry stdout | 2024-04-22 03:40:45,639 [275] [DEBUG] [app] Ending request: urn:request:5c96540d-2700-4818-8b63-9ad06ced1f8c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5c96540d-2700-4818-8b63-9ad06ced1f8c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:40:45,640 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:40:45,641 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:40:45,641 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:45,642 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:40:45,643 [286] [DEBUG] [app] Starting request: urn:request:af3846bd-433b-4f59-87d6-cfe44d3a999e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:45,643 [286] [DEBUG] [app] Ending request: urn:request:af3846bd-433b-4f59-87d6-cfe44d3a999e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af3846bd-433b-4f59-87d6-cfe44d3a999e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:45,644 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:40:45,645 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:40:45,645 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:45,646 [286] [DEBUG] [app] Starting request: urn:request:3c6efda4-fd20-4ebd-9f83-d80ee13b23e4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:40:45,646 [286] [DEBUG] [app] Ending request: urn:request:3c6efda4-fd20-4ebd-9f83-d80ee13b23e4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3c6efda4-fd20-4ebd-9f83-d80ee13b23e4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:40:45,646 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:40:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:40:45,647 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:40:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:40:45,647 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:40:45,660 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:45,660 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:40:45,673 [286] [DEBUG] [app] Ending request: urn:request:3b2f0c66-409f-40b3-8804-3f852665f75d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3b2f0c66-409f-40b3-8804-3f852665f75d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:45,673 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:40:45,673 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.042 118 0.042)
gunicorn-web stdout | 2024-04-22 03:40:45,682 [284] [DEBUG] [app] Ending request: urn:request:9df322a3-9eb1-4139-9674-70b8ac81df46 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9df322a3-9eb1-4139-9674-70b8ac81df46', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:40:45,682 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:40:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.052 118 0.052)
gunicorn-web stdout | 2024-04-22 03:40:45,683 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:40:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
globalpromstats stdout | 2024-04-22 03:40:45,811 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:40:46,544 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:40:47,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:40:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:14.067050+00:00 (in 27.000610 seconds)
securityworker stdout | 2024-04-22 03:40:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:17 UTC)" (scheduled at 2024-04-22 03:40:47.065745+00:00)
securityworker stdout | 2024-04-22 03:40:47,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:40:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:40:47,070 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:40:47,072 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:40:47,087 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:40:47,091 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:40:47,091 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:40:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:40:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:10.248668+00:00 (in 23.002265 seconds)
namespacegcworker stdout | 2024-04-22 03:40:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:47 UTC)" (scheduled at 2024-04-22 03:40:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:40:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:40:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 47, 246664), True, datetime.datetime(2024, 4, 22, 3, 40, 47, 246664), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:40:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:40:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:40:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:40:47,600 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:40:48,484 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:40:48,998 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:40:49,384 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:40:49,387 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:40:49,390 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:40:50,132 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:40:50,553 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:40:50,683 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:40:50,823 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:40:50,886 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:40:50,895 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:40:50,899 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:40:50,914 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:40:50,922 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:40:50,924 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:40:50,928 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:40:50,932 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:40:51,094 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:40:51,097 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:40:51,100 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:40:51,106 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:40:51,110 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:40:51,393 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:40:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:40:54.433683+00:00 (in 2.002139 seconds)
notificationworker stdout | 2024-04-22 03:40:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:02 UTC)" (scheduled at 2024-04-22 03:40:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:40:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:40:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 52, 431984), True, datetime.datetime(2024, 4, 22, 3, 40, 52, 431984), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:40:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:40:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:40:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:40:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:40:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:18.047111+00:00 (in 25.000988 seconds)
exportactionlogsworker stdout | 2024-04-22 03:40:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:53 UTC)" (scheduled at 2024-04-22 03:40:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:40:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:40:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 40, 53, 46367), True, datetime.datetime(2024, 4, 22, 3, 40, 53, 46367), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:40:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:40:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:40:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:40:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:40:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:02.430973+00:00 (in 7.996651 seconds)
notificationworker stdout | 2024-04-22 03:40:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:54 UTC)" (scheduled at 2024-04-22 03:40:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:40:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:40:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:41:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:40:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:40:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:28.130973+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2024-04-22 03:40:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:28 UTC)" (scheduled at 2024-04-22 03:40:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:40:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 40, 58, 131682), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:40:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:40:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:40:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:41:00,632 [282] [DEBUG] [app] Starting request: urn:request:80f3b32c-b5fd-4dcf-9ecf-d1c658755146 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:00,632 [285] [DEBUG] [app] Starting request: urn:request:05719ec9-40d6-4927-abe3-53adfd39e5c3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:41:00,636 [280] [DEBUG] [app] Starting request: urn:request:b3210d5e-7183-4c81-8d30-eda547b52228 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:00,636 [275] [DEBUG] [app] Starting request: urn:request:886448a7-a9ce-44bf-a5c5-a82be19aaa68 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:00,636 [275] [DEBUG] [app] Ending request: urn:request:886448a7-a9ce-44bf-a5c5-a82be19aaa68 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:886448a7-a9ce-44bf-a5c5-a82be19aaa68', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:41:00,636 [280] [DEBUG] [app] Ending request: urn:request:b3210d5e-7183-4c81-8d30-eda547b52228 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b3210d5e-7183-4c81-8d30-eda547b52228', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:41:00,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:41:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:41:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:00,639 [285] [DEBUG] [app] Starting request: urn:request:62960e41-9274-43b9-95d4-dba2fc806263 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:00,640 [285] [DEBUG] [app] Ending request: urn:request:62960e41-9274-43b9-95d4-dba2fc806263 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:62960e41-9274-43b9-95d4-dba2fc806263', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:00,640 [284] [DEBUG] [app] Starting request: urn:request:bb6b913d-ae28-431f-a167-c0d850bc8b6b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:41:00,640 [284] [DEBUG] [app] Ending request: urn:request:bb6b913d-ae28-431f-a167-c0d850bc8b6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb6b913d-ae28-431f-a167-c0d850bc8b6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:41:00,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:00,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:00,661 [285] [DEBUG] [app] Ending request: urn:request:05719ec9-40d6-4927-abe3-53adfd39e5c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:05719ec9-40d6-4927-abe3-53adfd39e5c3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:00,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:41:00,662 [282] [DEBUG] [app] Ending request: urn:request:80f3b32c-b5fd-4dcf-9ecf-d1c658755146 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:80f3b32c-b5fd-4dcf-9ecf-d1c658755146', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 03:41:00,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 03:41:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:41:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:31.842666+00:00 (in 29.999400 seconds)
gcworker stdout | 2024-04-22 03:41:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:31 UTC)" (scheduled at 2024-04-22 03:41:01.842666+00:00)
gcworker stdout | 2024-04-22 03:41:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:41:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:41:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:41:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:09.033206+00:00 (in 6.997106 seconds)
repositorygcworker stdout | 2024-04-22 03:41:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:02 UTC)" (scheduled at 2024-04-22 03:41:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:41:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:41:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:41:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:12.430973+00:00 (in 9.999533 seconds)
notificationworker stdout | 2024-04-22 03:41:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:12 UTC)" (scheduled at 2024-04-22 03:41:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:41:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:41:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 2, 431697), True, datetime.datetime(2024, 4, 22, 3, 41, 2, 431697), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:41:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:41:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:41:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:41:03,616 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:41:04,881 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:41:07,509 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:41:08,762 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
repositorygcworker stdout | 2024-04-22 03:41:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:41:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:02.035613+00:00 (in 53.001815 seconds)
repositorygcworker stdout | 2024-04-22 03:41:09,034 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:09 UTC)" (scheduled at 2024-04-22 03:41:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:41:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:41:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 9, 34193), True, datetime.datetime(2024, 4, 22, 3, 41, 9, 34193), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:41:09,051 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:41:09,051 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:41:09,051 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:09 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:41:09,055 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:41:09,100 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:41:09,756 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:41:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:41:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:40.238173+00:00 (in 29.999431 seconds)
autopruneworker stdout | 2024-04-22 03:41:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:40 UTC)" (scheduled at 2024-04-22 03:41:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:41:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757210247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:41:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:41:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:30.247852+00:00 (in 19.998709 seconds)
namespacegcworker stdout | 2024-04-22 03:41:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:10 UTC)" (scheduled at 2024-04-22 03:41:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:41:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:41:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:41:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:41:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:41:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:41:10,349 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:41:10,437 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:41:11,848 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:41:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:22.430973+00:00 (in 9.999489 seconds)
notificationworker stdout | 2024-04-22 03:41:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:22 UTC)" (scheduled at 2024-04-22 03:41:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:41:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:41:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 12, 431781), True, datetime.datetime(2024, 4, 22, 3, 41, 12, 431781), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:41:12,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:41:12,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:41:12,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:41:12,707 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:41:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:41:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:17.065745+00:00 (in 2.998024 seconds)
securityworker stdout | 2024-04-22 03:41:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:44 UTC)" (scheduled at 2024-04-22 03:41:14.067050+00:00)
securityworker stdout | 2024-04-22 03:41:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:41:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:41:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:41:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:41:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:41:14,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:41:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:41:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:41:14,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:41:14,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:41:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 36, 14, 78355), 1, 2])
securityworker stdout | 2024-04-22 03:41:14,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:41:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:41:14,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 36, 14, 78355), 1, 2])
securityworker stdout | 2024-04-22 03:41:14,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:41:14,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:41:14,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:14,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:41:14,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:41:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:41:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:41:14,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:14,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:41:14,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:41:15,134 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:41:15,454 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:41:15,632 [285] [DEBUG] [app] Starting request: urn:request:8a71f9b3-12ab-4d5c-a200-052cb7427470 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:15,632 [282] [DEBUG] [app] Starting request: urn:request:34dfa075-f042-4bd4-a76e-195dc7fa3b66 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:41:15,635 [275] [DEBUG] [app] Starting request: urn:request:be5e7046-ec95-48f9-874e-1917e2e17ab4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:15,635 [283] [DEBUG] [app] Starting request: urn:request:f42aec5b-e28a-410a-a48b-875ff3ddde29 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:15,635 [275] [DEBUG] [app] Ending request: urn:request:be5e7046-ec95-48f9-874e-1917e2e17ab4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:be5e7046-ec95-48f9-874e-1917e2e17ab4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:41:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:41:15,636 [283] [DEBUG] [app] Ending request: urn:request:f42aec5b-e28a-410a-a48b-875ff3ddde29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f42aec5b-e28a-410a-a48b-875ff3ddde29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:41:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:15,639 [286] [DEBUG] [app] Starting request: urn:request:b28ac5ad-5e81-4481-8826-7214d5b47539 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:15,639 [285] [DEBUG] [app] Starting request: urn:request:517b3bf0-2aaa-41ef-b9cd-08f73f4bc5b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:15,639 [285] [DEBUG] [app] Ending request: urn:request:517b3bf0-2aaa-41ef-b9cd-08f73f4bc5b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:517b3bf0-2aaa-41ef-b9cd-08f73f4bc5b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:15,639 [286] [DEBUG] [app] Ending request: urn:request:b28ac5ad-5e81-4481-8826-7214d5b47539 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b28ac5ad-5e81-4481-8826-7214d5b47539', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:41:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:15,660 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:15,661 [282] [DEBUG] [app] Ending request: urn:request:34dfa075-f042-4bd4-a76e-195dc7fa3b66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:34dfa075-f042-4bd4-a76e-195dc7fa3b66', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:41:15,672 [285] [DEBUG] [app] Ending request: urn:request:8a71f9b3-12ab-4d5c-a200-052cb7427470 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a71f9b3-12ab-4d5c-a200-052cb7427470', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:15,672 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:15,672 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.041 118 0.042)
globalpromstats stdout | 2024-04-22 03:41:15,847 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:41:16,584 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:41:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:41:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:44.067050+00:00 (in 27.000785 seconds)
securityworker stdout | 2024-04-22 03:41:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:47 UTC)" (scheduled at 2024-04-22 03:41:17.065745+00:00)
securityworker stdout | 2024-04-22 03:41:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:41:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:41:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:41:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:41:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:41:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:41:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:41:17,630 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:41:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:41:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:53.045554+00:00 (in 34.997898 seconds)
exportactionlogsworker stdout | 2024-04-22 03:41:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:18 UTC)" (scheduled at 2024-04-22 03:41:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:41:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:41:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:41:18,508 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:41:19,035 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:41:19,391 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:41:19,395 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:41:19,398 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:41:20,163 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:41:20,592 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:41:20,722 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:41:20,836 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:41:20,899 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:41:20,912 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:41:20,919 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:41:20,927 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:41:20,935 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:41:20,940 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:41:20,945 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:41:20,950 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:41:21,109 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:41:21,116 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:41:21,120 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:41:21,127 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:41:21,130 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:41:21,433 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:41:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:32.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 03:41:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:32 UTC)" (scheduled at 2024-04-22 03:41:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:41:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:41:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 22, 431799), True, datetime.datetime(2024, 4, 22, 3, 41, 22, 431799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:41:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:41:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:41:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:41:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:41:24,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:24.738365+00:00 (in 59.999307 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:41:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:24 UTC)" (scheduled at 2024-04-22 03:41:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:41:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:41:24,753 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:41:24,753 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:41:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:41:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:58.130973+00:00 (in 29.999452 seconds)
buildlogsarchiver stdout | 2024-04-22 03:41:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:58 UTC)" (scheduled at 2024-04-22 03:41:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:41:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 41, 28, 132115), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:41:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:41:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:41:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:41:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:31.933721+00:00 (in 2.002160 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:29 UTC)" (scheduled at 2024-04-22 03:41:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 29, 931829), True, datetime.datetime(2024, 4, 22, 3, 41, 29, 931829), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:41:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:41:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:41:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:47.245913+00:00 (in 16.997535 seconds)
namespacegcworker stdout | 2024-04-22 03:41:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:46:30 UTC)" (scheduled at 2024-04-22 03:41:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 03:41:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 41, 30, 248626), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:41:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 41, 30, 248626), True, datetime.datetime(2024, 4, 22, 3, 41, 30, 248626), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:41:30,271 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 41, 30, 248626), True, datetime.datetime(2024, 4, 22, 3, 41, 30, 248626), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 3, 41, 30, 248626), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:41:30,274 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:41:30,275 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:46:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:41:30,632 [282] [DEBUG] [app] Starting request: urn:request:2a294b1b-a26b-4460-856c-5cd9d4171102 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:30,632 [284] [DEBUG] [app] Starting request: urn:request:4206a732-73a5-4902-a473-98c583c18b44 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:30,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:41:30,635 [278] [DEBUG] [app] Starting request: urn:request:4aea49e9-50fc-416d-a621-0c4381d4c20a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:30,635 [279] [DEBUG] [app] Starting request: urn:request:79b9eb54-b245-4fb8-ad1c-fd2f734507b7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:30,635 [279] [DEBUG] [app] Ending request: urn:request:79b9eb54-b245-4fb8-ad1c-fd2f734507b7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:79b9eb54-b245-4fb8-ad1c-fd2f734507b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:41:30,636 [278] [DEBUG] [app] Ending request: urn:request:4aea49e9-50fc-416d-a621-0c4381d4c20a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4aea49e9-50fc-416d-a621-0c4381d4c20a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:41:30,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:41:30,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:30,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:30,638 [282] [DEBUG] [app] Starting request: urn:request:23469bdc-65e5-42c5-8fde-cb806716ce75 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:30,639 [282] [DEBUG] [app] Ending request: urn:request:23469bdc-65e5-42c5-8fde-cb806716ce75 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23469bdc-65e5-42c5-8fde-cb806716ce75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:30,639 [286] [DEBUG] [app] Starting request: urn:request:0e1b83f4-d258-42ad-ab22-9bd4a56e1ba4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:41:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:30,639 [286] [DEBUG] [app] Ending request: urn:request:0e1b83f4-d258-42ad-ab22-9bd4a56e1ba4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0e1b83f4-d258-42ad-ab22-9bd4a56e1ba4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:30,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:30,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:41:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:30,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:30,660 [284] [DEBUG] [app] Ending request: urn:request:4206a732-73a5-4902-a473-98c583c18b44 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4206a732-73a5-4902-a473-98c583c18b44', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:30,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:41:30,667 [282] [DEBUG] [app] Ending request: urn:request:2a294b1b-a26b-4460-856c-5cd9d4171102 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2a294b1b-a26b-4460-856c-5cd9d4171102', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:30,667 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:30,667 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.036)
gcworker stdout | 2024-04-22 03:41:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:41:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:01.842666+00:00 (in 29.999490 seconds)
gcworker stdout | 2024-04-22 03:41:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:01 UTC)" (scheduled at 2024-04-22 03:41:31.842666+00:00)
gcworker stdout | 2024-04-22 03:41:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:41:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:41:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:41:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:29.931002+00:00 (in 57.996701 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:41:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:31 UTC)" (scheduled at 2024-04-22 03:41:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:41:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:41:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:41:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:42.430973+00:00 (in 9.999416 seconds)
notificationworker stdout | 2024-04-22 03:41:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:42 UTC)" (scheduled at 2024-04-22 03:41:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:41:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:41:32,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 32, 431946), True, datetime.datetime(2024, 4, 22, 3, 41, 32, 431946), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:41:32,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:41:32,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:41:32,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:41:33,652 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:41:34,896 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:41:37,547 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:41:38,787 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:41:39,062 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:41:39,108 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:41:39,792 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:41:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:41:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:10.238173+00:00 (in 29.999467 seconds)
autopruneworker stdout | 2024-04-22 03:41:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:10 UTC)" (scheduled at 2024-04-22 03:41:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:41:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757240247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:41:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:41:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:41:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:41:40,374 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:41:40,474 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:41:41,884 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:41:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:52.430973+00:00 (in 9.999438 seconds)
notificationworker stdout | 2024-04-22 03:41:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:52 UTC)" (scheduled at 2024-04-22 03:41:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:41:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:41:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 42, 431807), True, datetime.datetime(2024, 4, 22, 3, 41, 42, 431807), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:41:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:41:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:41:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:41:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:41:42,728 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:41:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:41:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:47.065745+00:00 (in 2.998102 seconds)
securityworker stdout | 2024-04-22 03:41:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:14 UTC)" (scheduled at 2024-04-22 03:41:44.067050+00:00)
securityworker stdout | 2024-04-22 03:41:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:41:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:41:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:41:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:41:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:41:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:41:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:41:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:41:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:41:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:41:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 36, 44, 77690), 1, 2])
securityworker stdout | 2024-04-22 03:41:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:41:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:41:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 36, 44, 77690), 1, 2])
securityworker stdout | 2024-04-22 03:41:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:41:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:41:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:41:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:41:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:41:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:41:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:41:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:41:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:41:45,170 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:41:45,492 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:41:45,632 [285] [DEBUG] [app] Starting request: urn:request:2d2ec3bb-eb52-4d31-a74c-ed6932ab00a6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:45,632 [286] [DEBUG] [app] Starting request: urn:request:f81388dc-755f-434a-a39b-810f26c700f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:41:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:45,635 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:41:45,636 [275] [DEBUG] [app] Starting request: urn:request:7bea82f6-3b3f-468c-a66c-4bb5bef503a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:45,637 [275] [DEBUG] [app] Ending request: urn:request:7bea82f6-3b3f-468c-a66c-4bb5bef503a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7bea82f6-3b3f-468c-a66c-4bb5bef503a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:41:45,637 [280] [DEBUG] [app] Starting request: urn:request:357f5a11-70c1-44f9-a455-75cb0988f6a3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:41:45,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:41:45,637 [280] [DEBUG] [app] Ending request: urn:request:357f5a11-70c1-44f9-a455-75cb0988f6a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:357f5a11-70c1-44f9-a455-75cb0988f6a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:41:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:41:45,638 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:45,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:45,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:45,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:41:45,641 [286] [DEBUG] [app] Starting request: urn:request:19d5cd9d-6906-408c-a8e7-4648e5e79d11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:45,641 [286] [DEBUG] [app] Ending request: urn:request:19d5cd9d-6906-408c-a8e7-4648e5e79d11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:19d5cd9d-6906-408c-a8e7-4648e5e79d11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:45,641 [285] [DEBUG] [app] Starting request: urn:request:2afee394-5208-420e-a99d-f6c6e31d9e69 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:41:45,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:45,641 [285] [DEBUG] [app] Ending request: urn:request:2afee394-5208-420e-a99d-f6c6e31d9e69 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2afee394-5208-420e-a99d-f6c6e31d9e69', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:41:45,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:41:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:41:45,642 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:41:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:41:45,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:41:45,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:45,643 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:41:45,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:45,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:41:45,664 [285] [DEBUG] [app] Ending request: urn:request:2d2ec3bb-eb52-4d31-a74c-ed6932ab00a6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d2ec3bb-eb52-4d31-a74c-ed6932ab00a6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:45,664 [286] [DEBUG] [app] Ending request: urn:request:f81388dc-755f-434a-a39b-810f26c700f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f81388dc-755f-434a-a39b-810f26c700f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:41:45,664 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:45,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:41:45,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:41:45,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:41:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.033)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:41:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
globalpromstats stdout | 2024-04-22 03:41:45,883 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:41:46,620 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:41:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:41:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:14.067050+00:00 (in 27.000771 seconds)
securityworker stdout | 2024-04-22 03:41:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:17 UTC)" (scheduled at 2024-04-22 03:41:47.065745+00:00)
securityworker stdout | 2024-04-22 03:41:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:41:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:41:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:41:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:41:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:41:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:41:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:41:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:41:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:10.248668+00:00 (in 23.002263 seconds)
namespacegcworker stdout | 2024-04-22 03:41:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:47 UTC)" (scheduled at 2024-04-22 03:41:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:41:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:41:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 47, 246716), True, datetime.datetime(2024, 4, 22, 3, 41, 47, 246716), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:41:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:41:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:41:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:41:47,667 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:41:48,548 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:41:49,073 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:41:49,404 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:41:49,409 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:41:49,413 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:41:50,196 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:41:50,619 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:41:50,735 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:41:50,847 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:41:50,906 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:41:50,920 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:41:50,930 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:41:50,934 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:41:50,946 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:41:50,949 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:41:50,952 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:41:50,958 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:41:51,118 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:41:51,124 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:41:51,127 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:41:51,133 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:41:51,138 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:41:51,462 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:41:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:41:54.433683+00:00 (in 2.002267 seconds)
notificationworker stdout | 2024-04-22 03:41:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:02 UTC)" (scheduled at 2024-04-22 03:41:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:41:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:41:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 52, 431690), True, datetime.datetime(2024, 4, 22, 3, 41, 52, 431690), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:41:52,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:41:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:41:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:41:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:41:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:18.047111+00:00 (in 25.000954 seconds)
exportactionlogsworker stdout | 2024-04-22 03:41:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:53 UTC)" (scheduled at 2024-04-22 03:41:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:41:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:41:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 41, 53, 46585), True, datetime.datetime(2024, 4, 22, 3, 41, 53, 46585), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:41:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:41:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:41:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:41:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:41:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:02.430973+00:00 (in 7.996811 seconds)
notificationworker stdout | 2024-04-22 03:41:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:54 UTC)" (scheduled at 2024-04-22 03:41:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:41:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:41:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:42:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:41:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:41:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:28.130973+00:00 (in 29.999335 seconds)
buildlogsarchiver stdout | 2024-04-22 03:41:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:28 UTC)" (scheduled at 2024-04-22 03:41:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:41:58,133 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 41, 58, 132116), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:41:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:41:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:41:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:42:00,632 [282] [DEBUG] [app] Starting request: urn:request:99e2e70d-5ed3-483f-a653-d0322c1b796b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:00,632 [286] [DEBUG] [app] Starting request: urn:request:4a9d6319-b342-4371-a90a-1ae9740a0ecb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:00,635 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:42:00,636 [275] [DEBUG] [app] Starting request: urn:request:f4029dbd-fa67-477a-bd73-11bafa2a8852 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:00,636 [275] [DEBUG] [app] Ending request: urn:request:f4029dbd-fa67-477a-bd73-11bafa2a8852 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4029dbd-fa67-477a-bd73-11bafa2a8852', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:42:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:42:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:42:00,637 [283] [DEBUG] [app] Starting request: urn:request:1b57d3c7-f97f-424a-b2ee-e8f480d27142 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:00,638 [283] [DEBUG] [app] Ending request: urn:request:1b57d3c7-f97f-424a-b2ee-e8f480d27142 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1b57d3c7-f97f-424a-b2ee-e8f480d27142', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:42:00,638 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:00,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:00,640 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:00,641 [284] [DEBUG] [app] Starting request: urn:request:03de3d25-c47c-42bf-b1fe-d991a7c1ca96 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:00,642 [284] [DEBUG] [app] Ending request: urn:request:03de3d25-c47c-42bf-b1fe-d991a7c1ca96 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:03de3d25-c47c-42bf-b1fe-d991a7c1ca96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:42:00,642 [285] [DEBUG] [app] Starting request: urn:request:2cbbad60-8987-4e97-ac69-2d0c48e1c8f9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:42:00,642 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:00,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:00,643 [285] [DEBUG] [app] Ending request: urn:request:2cbbad60-8987-4e97-ac69-2d0c48e1c8f9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2cbbad60-8987-4e97-ac69-2d0c48e1c8f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:42:00,643 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:00,643 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:42:00,643 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:00,644 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:00,653 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:00,653 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:00,664 [282] [DEBUG] [app] Ending request: urn:request:99e2e70d-5ed3-483f-a653-d0322c1b796b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99e2e70d-5ed3-483f-a653-d0322c1b796b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:00,665 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:00,665 [286] [DEBUG] [app] Ending request: urn:request:4a9d6319-b342-4371-a90a-1ae9740a0ecb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4a9d6319-b342-4371-a90a-1ae9740a0ecb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:00,665 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:00,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:42:00,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gcworker stdout | 2024-04-22 03:42:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:42:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:31.842666+00:00 (in 29.999484 seconds)
gcworker stdout | 2024-04-22 03:42:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:31 UTC)" (scheduled at 2024-04-22 03:42:01.842666+00:00)
gcworker stdout | 2024-04-22 03:42:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:42:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:42:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:42:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:09.033206+00:00 (in 6.997025 seconds)
repositorygcworker stdout | 2024-04-22 03:42:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:02 UTC)" (scheduled at 2024-04-22 03:42:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:42:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:42:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:42:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:12.430973+00:00 (in 9.999527 seconds)
notificationworker stdout | 2024-04-22 03:42:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:12 UTC)" (scheduled at 2024-04-22 03:42:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:42:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:42:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 2, 431746), True, datetime.datetime(2024, 4, 22, 3, 42, 2, 431746), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:42:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:42:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:42:03,689 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:42:04,928 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:42:07,586 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:42:08,795 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
repositorygcworker stdout | 2024-04-22 03:42:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:42:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:02.035613+00:00 (in 53.001832 seconds)
repositorygcworker stdout | 2024-04-22 03:42:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:09 UTC)" (scheduled at 2024-04-22 03:42:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:42:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:42:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 9, 34142), True, datetime.datetime(2024, 4, 22, 3, 42, 9, 34142), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:42:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:42:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:42:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:09 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:42:09,098 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:42:09,138 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:42:09,818 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:42:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:42:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:40.238173+00:00 (in 29.999549 seconds)
autopruneworker stdout | 2024-04-22 03:42:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:40 UTC)" (scheduled at 2024-04-22 03:42:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:42:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757270247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:42:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:42:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:47.245913+00:00 (in 36.996780 seconds)
namespacegcworker stdout | 2024-04-22 03:42:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:10 UTC)" (scheduled at 2024-04-22 03:42:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:42:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:42:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:42:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:42:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:42:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:42:10,389 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:42:10,511 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:42:11,892 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:42:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:22.430973+00:00 (in 9.999421 seconds)
notificationworker stdout | 2024-04-22 03:42:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:22 UTC)" (scheduled at 2024-04-22 03:42:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:42:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:42:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 12, 431986), True, datetime.datetime(2024, 4, 22, 3, 42, 12, 431986), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:42:12,461 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:42:12,461 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:12,461 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:42:12,764 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:42:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:42:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:17.065745+00:00 (in 2.998209 seconds)
securityworker stdout | 2024-04-22 03:42:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:44 UTC)" (scheduled at 2024-04-22 03:42:14.067050+00:00)
securityworker stdout | 2024-04-22 03:42:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:42:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:42:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:42:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:42:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:42:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:42:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:42:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:42:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:42:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:42:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 37, 14, 76971), 1, 2])
securityworker stdout | 2024-04-22 03:42:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:42:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:42:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 37, 14, 76971), 1, 2])
securityworker stdout | 2024-04-22 03:42:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:42:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:42:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:42:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:42:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:42:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:42:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:42:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:42:15,200 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:42:15,522 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:42:15,633 [286] [DEBUG] [app] Starting request: urn:request:43919f0d-5825-4f1d-9878-1ad632a26221 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:15,633 [285] [DEBUG] [app] Starting request: urn:request:543c641c-9c6e-4e21-a19d-c00d01bc0ce7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:42:15,636 [275] [DEBUG] [app] Starting request: urn:request:c3dfacd1-9bd2-4ca6-9bfd-e0e9c770a582 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:15,636 [283] [DEBUG] [app] Starting request: urn:request:b75fed44-9d1b-4b0d-bfe2-bce027f95c6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:15,636 [275] [DEBUG] [app] Ending request: urn:request:c3dfacd1-9bd2-4ca6-9bfd-e0e9c770a582 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3dfacd1-9bd2-4ca6-9bfd-e0e9c770a582', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:42:15,636 [283] [DEBUG] [app] Ending request: urn:request:b75fed44-9d1b-4b0d-bfe2-bce027f95c6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b75fed44-9d1b-4b0d-bfe2-bce027f95c6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:42:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:42:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:42:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:15,639 [286] [DEBUG] [app] Starting request: urn:request:60b62561-20b5-4df6-ae9e-c0086133a5e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:15,639 [286] [DEBUG] [app] Ending request: urn:request:60b62561-20b5-4df6-ae9e-c0086133a5e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:60b62561-20b5-4df6-ae9e-c0086133a5e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:42:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:15,640 [284] [DEBUG] [app] Starting request: urn:request:8b4fea8b-0310-489e-9d63-cd5545b37369 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:15,640 [284] [DEBUG] [app] Ending request: urn:request:8b4fea8b-0310-489e-9d63-cd5545b37369 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b4fea8b-0310-489e-9d63-cd5545b37369', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:42:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:15,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:42:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:15,660 [286] [DEBUG] [app] Ending request: urn:request:43919f0d-5825-4f1d-9878-1ad632a26221 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43919f0d-5825-4f1d-9878-1ad632a26221', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:42:15,661 [285] [DEBUG] [app] Ending request: urn:request:543c641c-9c6e-4e21-a19d-c00d01bc0ce7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:543c641c-9c6e-4e21-a19d-c00d01bc0ce7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:15,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:15,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
globalpromstats stdout | 2024-04-22 03:42:15,920 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:42:16,657 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:42:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:42:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:44.067050+00:00 (in 27.000825 seconds)
securityworker stdout | 2024-04-22 03:42:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:47 UTC)" (scheduled at 2024-04-22 03:42:17.065745+00:00)
securityworker stdout | 2024-04-22 03:42:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:42:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:42:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:42:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:42:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:42:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:42:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:42:17,686 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:42:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:42:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:23.046900+00:00 (in 4.999209 seconds)
exportactionlogsworker stdout | 2024-04-22 03:42:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:18 UTC)" (scheduled at 2024-04-22 03:42:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:42:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:42:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:42:18,574 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:42:19,099 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:42:19,412 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:42:19,417 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:42:19,420 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:42:20,233 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:42:20,650 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:42:20,767 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:42:20,857 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:42:20,916 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:42:20,927 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:42:20,940 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:42:20,943 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:42:20,954 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:42:20,962 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:42:20,965 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:42:20,970 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:42:21,127 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:42:21,131 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:42:21,134 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:42:21,138 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:42:21,145 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:42:21,498 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:42:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:32.430973+00:00 (in 9.999478 seconds)
notificationworker stdout | 2024-04-22 03:42:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:32 UTC)" (scheduled at 2024-04-22 03:42:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:42:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:42:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 22, 431730), True, datetime.datetime(2024, 4, 22, 3, 42, 22, 431730), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:42:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:42:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:32 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:42:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:42:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:53.045554+00:00 (in 29.998114 seconds)
exportactionlogsworker stdout | 2024-04-22 03:42:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:47:23 UTC)" (scheduled at 2024-04-22 03:42:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 03:42:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 42, 23, 47850), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:42:23,062 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 42, 23, 47850), True, datetime.datetime(2024, 4, 22, 3, 42, 23, 47850), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:42:23,066 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 42, 23, 47850), True, datetime.datetime(2024, 4, 22, 3, 42, 23, 47850), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 3, 42, 23, 47850), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:42:23,070 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:42:23,070 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:47:23 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:42:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:42:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:24.738365+00:00 (in 59.999433 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:42:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:24 UTC)" (scheduled at 2024-04-22 03:42:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:42:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:42:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:42:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:42:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:42:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:58.130973+00:00 (in 29.999508 seconds)
buildlogsarchiver stdout | 2024-04-22 03:42:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:58 UTC)" (scheduled at 2024-04-22 03:42:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:42:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 42, 28, 131798), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:42:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:42:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:42:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:42:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:31.933721+00:00 (in 2.002114 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:29 UTC)" (scheduled at 2024-04-22 03:42:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 29, 931955), True, datetime.datetime(2024, 4, 22, 3, 42, 29, 931955), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:42:29,948 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:42:30,632 [282] [DEBUG] [app] Starting request: urn:request:76fe46b5-ec17-42ed-b54e-522a835a7063 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:30,632 [286] [DEBUG] [app] Starting request: urn:request:4f663074-4d99-43fe-8339-35cb8980df87 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:42:30,635 [275] [DEBUG] [app] Starting request: urn:request:2e7d1199-539d-491c-a385-acbcc2e7c9b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:30,635 [280] [DEBUG] [app] Starting request: urn:request:6e345d4b-47c3-4e99-a37e-8993efca1924 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:30,636 [280] [DEBUG] [app] Ending request: urn:request:6e345d4b-47c3-4e99-a37e-8993efca1924 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6e345d4b-47c3-4e99-a37e-8993efca1924', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:42:30,636 [275] [DEBUG] [app] Ending request: urn:request:2e7d1199-539d-491c-a385-acbcc2e7c9b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2e7d1199-539d-491c-a385-acbcc2e7c9b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:42:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:42:30,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:42:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:30,639 [286] [DEBUG] [app] Starting request: urn:request:d5eb29a7-6d24-4644-b209-31e3a6085462 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:30,639 [285] [DEBUG] [app] Starting request: urn:request:66ad3095-49ea-4a50-8a05-7b6f844b3d8b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:30,639 [286] [DEBUG] [app] Ending request: urn:request:d5eb29a7-6d24-4644-b209-31e3a6085462 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5eb29a7-6d24-4644-b209-31e3a6085462', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:42:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:30,640 [285] [DEBUG] [app] Ending request: urn:request:66ad3095-49ea-4a50-8a05-7b6f844b3d8b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:66ad3095-49ea-4a50-8a05-7b6f844b3d8b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:42:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:30,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:42:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:30,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:30,661 [286] [DEBUG] [app] Ending request: urn:request:4f663074-4d99-43fe-8339-35cb8980df87 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4f663074-4d99-43fe-8339-35cb8980df87', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:30,661 [282] [DEBUG] [app] Ending request: urn:request:76fe46b5-ec17-42ed-b54e-522a835a7063 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76fe46b5-ec17-42ed-b54e-522a835a7063', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:42:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:42:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:42:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:01.842666+00:00 (in 29.999487 seconds)
gcworker stdout | 2024-04-22 03:42:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:01 UTC)" (scheduled at 2024-04-22 03:42:31.842666+00:00)
gcworker stdout | 2024-04-22 03:42:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:42:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:42:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:42:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:29.931002+00:00 (in 57.996731 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:42:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:31 UTC)" (scheduled at 2024-04-22 03:42:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:42:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:42:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:42:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:38.433394+00:00 (in 6.001938 seconds)
notificationworker stdout | 2024-04-22 03:42:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:42 UTC)" (scheduled at 2024-04-22 03:42:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:42:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:42:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 32, 431731), True, datetime.datetime(2024, 4, 22, 3, 42, 32, 431731), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:42:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:42:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:42:33,730 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:42:34,965 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:42:37,621 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
notificationworker stdout | 2024-04-22 03:42:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:38,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:42.430973+00:00 (in 3.996982 seconds)
notificationworker stdout | 2024-04-22 03:42:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:47:38 UTC)" (scheduled at 2024-04-22 03:42:38.433394+00:00)
notificationworker stdout | 2024-04-22 03:42:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 42, 38, 434308), 'notification/%'])
notificationworker stdout | 2024-04-22 03:42:38,449 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 42, 38, 434308), True, datetime.datetime(2024, 4, 22, 3, 42, 38, 434308), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 03:42:38,453 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 42, 38, 434308), True, datetime.datetime(2024, 4, 22, 3, 42, 38, 434308), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 3, 42, 38, 434308), 'notification/%'])
notificationworker stdout | 2024-04-22 03:42:38,457 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:38,457 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:47:38 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:42:38,831 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:42:39,135 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:42:39,170 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:42:39,855 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:42:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:42:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:10.238173+00:00 (in 29.999444 seconds)
autopruneworker stdout | 2024-04-22 03:42:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:10 UTC)" (scheduled at 2024-04-22 03:42:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:42:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757300247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:42:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:42:40,255 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:42:40,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:42:40,398 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:42:40,519 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:42:41,929 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:42:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:52.430973+00:00 (in 9.999431 seconds)
notificationworker stdout | 2024-04-22 03:42:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:52 UTC)" (scheduled at 2024-04-22 03:42:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:42:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:42:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 42, 431789), True, datetime.datetime(2024, 4, 22, 3, 42, 42, 431789), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:42:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:42:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:42:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:42:42,777 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:42:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:42:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:47.065745+00:00 (in 2.998239 seconds)
securityworker stdout | 2024-04-22 03:42:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:14 UTC)" (scheduled at 2024-04-22 03:42:44.067050+00:00)
securityworker stdout | 2024-04-22 03:42:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:42:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:42:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:42:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:42:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:42:44,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:42:44,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:42:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:42:44,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:42:44,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:42:44,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 37, 44, 78702), 1, 2])
securityworker stdout | 2024-04-22 03:42:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:42:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:42:44,103 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 37, 44, 78702), 1, 2])
securityworker stdout | 2024-04-22 03:42:44,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:42:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:42:44,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:44,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:42:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:42:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:42:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:42:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:42:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:42:44,109 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:42:45,236 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:42:45,559 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:42:45,632 [282] [DEBUG] [app] Starting request: urn:request:f67c42ed-a3c8-40ed-bea0-474e4d4f2767 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:45,632 [285] [DEBUG] [app] Starting request: urn:request:972b0848-446e-4017-95e3-7b7e74ce1b20 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:42:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:42:45,635 [279] [DEBUG] [app] Starting request: urn:request:2d7a4f75-259c-4ae8-b322-e19eef09d2fa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:45,635 [280] [DEBUG] [app] Starting request: urn:request:aecfb1da-4400-491d-b329-440105efde75 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:42:45,635 [280] [DEBUG] [app] Ending request: urn:request:aecfb1da-4400-491d-b329-440105efde75 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aecfb1da-4400-491d-b329-440105efde75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:42:45,635 [279] [DEBUG] [app] Ending request: urn:request:2d7a4f75-259c-4ae8-b322-e19eef09d2fa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2d7a4f75-259c-4ae8-b322-e19eef09d2fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:42:45,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:42:45,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:42:45,638 [282] [DEBUG] [app] Starting request: urn:request:245b5240-f75c-47dd-95ee-f37270f12492 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:45,638 [282] [DEBUG] [app] Ending request: urn:request:245b5240-f75c-47dd-95ee-f37270f12492 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:245b5240-f75c-47dd-95ee-f37270f12492', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:42:45,638 [284] [DEBUG] [app] Starting request: urn:request:a5601193-3c66-41f5-b669-1dcd691be989 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:42:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:42:45,639 [284] [DEBUG] [app] Ending request: urn:request:a5601193-3c66-41f5-b669-1dcd691be989 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a5601193-3c66-41f5-b669-1dcd691be989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:42:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:42:45,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:42:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:42:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:42:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:42:45,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:45,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:42:45,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:42:45,659 [285] [DEBUG] [app] Ending request: urn:request:972b0848-446e-4017-95e3-7b7e74ce1b20 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:972b0848-446e-4017-95e3-7b7e74ce1b20', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:45,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:42:45,660 [282] [DEBUG] [app] Ending request: urn:request:f67c42ed-a3c8-40ed-bea0-474e4d4f2767 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f67c42ed-a3c8-40ed-bea0-474e4d4f2767', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:42:45,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:42:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.029)
gunicorn-web stdout | 2024-04-22 03:42:45,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:42:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:42:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
globalpromstats stdout | 2024-04-22 03:42:45,956 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:42:46,686 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:42:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:42:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:14.067050+00:00 (in 27.000719 seconds)
securityworker stdout | 2024-04-22 03:42:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:17 UTC)" (scheduled at 2024-04-22 03:42:47.065745+00:00)
securityworker stdout | 2024-04-22 03:42:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:42:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:42:47,070 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:42:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:42:47,084 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:42:47,088 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:42:47,088 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:42:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:42:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:10.248668+00:00 (in 23.002260 seconds)
namespacegcworker stdout | 2024-04-22 03:42:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:47 UTC)" (scheduled at 2024-04-22 03:42:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:42:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:42:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 47, 246686), True, datetime.datetime(2024, 4, 22, 3, 42, 47, 246686), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:42:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:42:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:42:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:42:47,726 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:42:48,613 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:42:49,137 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:42:49,428 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:42:49,431 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:42:49,439 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:42:50,270 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:42:50,673 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:42:50,787 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:42:50,868 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:42:50,930 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:42:50,939 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:42:50,953 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:42:50,957 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:42:50,967 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:42:50,972 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:42:50,975 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:42:50,979 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:42:51,140 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:42:51,146 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:42:51,150 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:42:51,153 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:42:51,155 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:42:51,534 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:42:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:42:54.433683+00:00 (in 2.002234 seconds)
notificationworker stdout | 2024-04-22 03:42:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:02 UTC)" (scheduled at 2024-04-22 03:42:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:42:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:42:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 52, 431754), True, datetime.datetime(2024, 4, 22, 3, 42, 52, 431754), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:42:52,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:42:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:42:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:42:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:42:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:18.047111+00:00 (in 25.001054 seconds)
exportactionlogsworker stdout | 2024-04-22 03:42:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:53 UTC)" (scheduled at 2024-04-22 03:42:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:42:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:42:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 42, 53, 46355), True, datetime.datetime(2024, 4, 22, 3, 42, 53, 46355), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:42:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:42:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:42:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:42:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:42:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:02.430973+00:00 (in 7.996771 seconds)
notificationworker stdout | 2024-04-22 03:42:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:54 UTC)" (scheduled at 2024-04-22 03:42:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:42:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:42:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:43:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:42:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:42:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:28.130973+00:00 (in 29.999477 seconds)
buildlogsarchiver stdout | 2024-04-22 03:42:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:28 UTC)" (scheduled at 2024-04-22 03:42:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:42:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 42, 58, 131871), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:42:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:42:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:42:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:43:00,632 [282] [DEBUG] [app] Starting request: urn:request:f368e153-d8a6-46c7-a0dc-fa70cf3fb81e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:00,632 [284] [DEBUG] [app] Starting request: urn:request:23d31c48-383f-4974-beca-38341eb89437 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:43:00,636 [275] [DEBUG] [app] Starting request: urn:request:5459fbf5-ccca-450a-89da-73c8f377a737 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:00,636 [277] [DEBUG] [app] Starting request: urn:request:fbcbd7bd-b7ec-46b5-9e53-22f3ac90c7dd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:00,636 [275] [DEBUG] [app] Ending request: urn:request:5459fbf5-ccca-450a-89da-73c8f377a737 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5459fbf5-ccca-450a-89da-73c8f377a737', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:00,636 [277] [DEBUG] [app] Ending request: urn:request:fbcbd7bd-b7ec-46b5-9e53-22f3ac90c7dd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fbcbd7bd-b7ec-46b5-9e53-22f3ac90c7dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:43:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:43:00,637 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:00,640 [286] [DEBUG] [app] Starting request: urn:request:41770dcd-eb53-4022-a42a-cf1283154fd0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:00,640 [285] [DEBUG] [app] Starting request: urn:request:27ab6100-b0e0-44a4-87d5-489e71769d79 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:00,640 [286] [DEBUG] [app] Ending request: urn:request:41770dcd-eb53-4022-a42a-cf1283154fd0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:41770dcd-eb53-4022-a42a-cf1283154fd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:00,641 [285] [DEBUG] [app] Ending request: urn:request:27ab6100-b0e0-44a4-87d5-489e71769d79 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:27ab6100-b0e0-44a4-87d5-489e71769d79', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:43:00,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:43:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:00,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:00,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:00,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:00,663 [284] [DEBUG] [app] Ending request: urn:request:23d31c48-383f-4974-beca-38341eb89437 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23d31c48-383f-4974-beca-38341eb89437', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:00,663 [282] [DEBUG] [app] Ending request: urn:request:f368e153-d8a6-46c7-a0dc-fa70cf3fb81e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f368e153-d8a6-46c7-a0dc-fa70cf3fb81e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:00,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:00,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:00,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:43:00,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gcworker stdout | 2024-04-22 03:43:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:43:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:31.842666+00:00 (in 29.999500 seconds)
gcworker stdout | 2024-04-22 03:43:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:31 UTC)" (scheduled at 2024-04-22 03:43:01.842666+00:00)
gcworker stdout | 2024-04-22 03:43:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:43:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:43:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:43:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:09.033206+00:00 (in 6.997037 seconds)
repositorygcworker stdout | 2024-04-22 03:43:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:02 UTC)" (scheduled at 2024-04-22 03:43:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:43:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:43:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:43:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:12.430973+00:00 (in 9.999469 seconds)
notificationworker stdout | 2024-04-22 03:43:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:12 UTC)" (scheduled at 2024-04-22 03:43:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:43:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:43:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 2, 431826), True, datetime.datetime(2024, 4, 22, 3, 43, 2, 431826), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:43:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:43:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:43:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:43:03,767 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:43:05,001 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:43:07,647 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:43:08,868 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
repositorygcworker stdout | 2024-04-22 03:43:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:43:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:26.035364+00:00 (in 17.001735 seconds)
repositorygcworker stdout | 2024-04-22 03:43:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:09 UTC)" (scheduled at 2024-04-22 03:43:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:43:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:43:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 9, 33898), True, datetime.datetime(2024, 4, 22, 3, 43, 9, 33898), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:43:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:43:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:43:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:09 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:43:09,172 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:43:09,185 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:43:09,892 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:43:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:43:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:40.238173+00:00 (in 29.999502 seconds)
autopruneworker stdout | 2024-04-22 03:43:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:40 UTC)" (scheduled at 2024-04-22 03:43:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:43:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757330247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:43:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:43:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:47.245913+00:00 (in 36.996747 seconds)
namespacegcworker stdout | 2024-04-22 03:43:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:10 UTC)" (scheduled at 2024-04-22 03:43:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:43:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:43:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:43:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:43:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:43:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:43:10,413 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:43:10,555 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:43:11,966 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:43:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:22.430973+00:00 (in 9.999498 seconds)
notificationworker stdout | 2024-04-22 03:43:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:22 UTC)" (scheduled at 2024-04-22 03:43:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:43:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:43:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 12, 431736), True, datetime.datetime(2024, 4, 22, 3, 43, 12, 431736), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:43:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:43:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:43:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:43:12,804 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:43:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:43:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:17.065745+00:00 (in 2.998127 seconds)
securityworker stdout | 2024-04-22 03:43:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:44 UTC)" (scheduled at 2024-04-22 03:43:14.067050+00:00)
securityworker stdout | 2024-04-22 03:43:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:43:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:43:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:43:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:43:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:43:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:43:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:43:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:43:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:43:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:43:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 38, 14, 77708), 1, 2])
securityworker stdout | 2024-04-22 03:43:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:43:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:43:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 38, 14, 77708), 1, 2])
securityworker stdout | 2024-04-22 03:43:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:43:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:43:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:43:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:43:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:43:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:43:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:43:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:43:15,273 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:43:15,566 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:43:15,632 [282] [DEBUG] [app] Starting request: urn:request:6ff420d6-ab60-4abb-94f9-2f3232f26248 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:15,632 [286] [DEBUG] [app] Starting request: urn:request:d328a1bc-9f23-4112-acad-a90f53eef078 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:43:15,635 [281] [DEBUG] [app] Starting request: urn:request:006c1c9a-df4a-4f0b-a4a2-eee8232ed771 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:15,635 [275] [DEBUG] [app] Starting request: urn:request:4fc0a324-cbe4-4c29-a702-3002b38412a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:15,635 [281] [DEBUG] [app] Ending request: urn:request:006c1c9a-df4a-4f0b-a4a2-eee8232ed771 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:006c1c9a-df4a-4f0b-a4a2-eee8232ed771', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:15,635 [275] [DEBUG] [app] Ending request: urn:request:4fc0a324-cbe4-4c29-a702-3002b38412a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4fc0a324-cbe4-4c29-a702-3002b38412a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:43:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:43:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:43:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:15,638 [286] [DEBUG] [app] Starting request: urn:request:f48e93f7-c263-4bce-8a8e-0e411839a11d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:15,638 [282] [DEBUG] [app] Starting request: urn:request:d5cccac2-28ec-471e-8156-aa892dab13c5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:15,638 [286] [DEBUG] [app] Ending request: urn:request:f48e93f7-c263-4bce-8a8e-0e411839a11d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f48e93f7-c263-4bce-8a8e-0e411839a11d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:15,639 [282] [DEBUG] [app] Ending request: urn:request:d5cccac2-28ec-471e-8156-aa892dab13c5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d5cccac2-28ec-471e-8156-aa892dab13c5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:43:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:15,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:15,660 [286] [DEBUG] [app] Ending request: urn:request:d328a1bc-9f23-4112-acad-a90f53eef078 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d328a1bc-9f23-4112-acad-a90f53eef078', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:43:15,666 [282] [DEBUG] [app] Ending request: urn:request:6ff420d6-ab60-4abb-94f9-2f3232f26248 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ff420d6-ab60-4abb-94f9-2f3232f26248', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:15,666 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:15,666 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.035)
globalpromstats stdout | 2024-04-22 03:43:15,976 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:43:16,723 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:43:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:43:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:44.067050+00:00 (in 27.000774 seconds)
securityworker stdout | 2024-04-22 03:43:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:47 UTC)" (scheduled at 2024-04-22 03:43:17.065745+00:00)
securityworker stdout | 2024-04-22 03:43:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:43:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:43:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:43:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:43:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:43:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:43:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:43:17,762 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:43:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:43:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:53.045554+00:00 (in 34.997910 seconds)
exportactionlogsworker stdout | 2024-04-22 03:43:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:18 UTC)" (scheduled at 2024-04-22 03:43:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:43:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:43:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:43:18,636 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:43:19,148 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:43:19,438 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:43:19,442 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:43:19,446 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:43:20,303 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:43:20,710 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:43:20,804 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:43:20,877 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:43:20,941 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:43:20,951 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:43:20,962 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:43:20,967 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:43:20,978 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:43:20,981 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:43:20,986 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:43:20,990 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:43:21,148 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:43:21,154 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:43:21,159 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:43:21,162 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:43:21,165 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:43:21,571 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:43:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:32.430973+00:00 (in 9.999523 seconds)
notificationworker stdout | 2024-04-22 03:43:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:32 UTC)" (scheduled at 2024-04-22 03:43:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:43:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:43:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 22, 431743), True, datetime.datetime(2024, 4, 22, 3, 43, 22, 431743), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:43:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:43:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:43:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:43:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:43:24,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:24.738365+00:00 (in 59.999103 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:43:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:24 UTC)" (scheduled at 2024-04-22 03:43:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:43:24,740 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:43:24,758 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:43:24,758 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:43:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:43:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:02.035613+00:00 (in 35.999793 seconds)
repositorygcworker stdout | 2024-04-22 03:43:26,035 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:48:26 UTC)" (scheduled at 2024-04-22 03:43:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 03:43:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 43, 26, 36021), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:43:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 43, 26, 36021), True, datetime.datetime(2024, 4, 22, 3, 43, 26, 36021), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:43:26,054 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 43, 26, 36021), True, datetime.datetime(2024, 4, 22, 3, 43, 26, 36021), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 3, 43, 26, 36021), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:43:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:43:26,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:48:26 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:43:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:43:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:58.130973+00:00 (in 29.999098 seconds)
buildlogsarchiver stdout | 2024-04-22 03:43:28,132 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:58 UTC)" (scheduled at 2024-04-22 03:43:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:43:28,133 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 43, 28, 132368), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:43:28,149 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:43:28,150 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:43:28,150 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:43:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:30.933445+00:00 (in 1.001952 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:29 UTC)" (scheduled at 2024-04-22 03:43:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 29, 931749), True, datetime.datetime(2024, 4, 22, 3, 43, 29, 931749), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:43:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:43:30,633 [284] [DEBUG] [app] Starting request: urn:request:3c6e5968-b013-49bf-a687-61d3f2e7c557 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:30,633 [282] [DEBUG] [app] Starting request: urn:request:16ee3347-dfe2-47e7-8e5f-7628a30b024d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:30,635 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:30,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:43:30,638 [276] [DEBUG] [app] Starting request: urn:request:19189f47-595f-41c2-99f9-77cd510a3745 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:30,638 [279] [DEBUG] [app] Starting request: urn:request:868ea2a6-c55e-4cac-b3ba-2939059496f7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:30,638 [276] [DEBUG] [app] Ending request: urn:request:19189f47-595f-41c2-99f9-77cd510a3745 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:19189f47-595f-41c2-99f9-77cd510a3745', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:30,638 [279] [DEBUG] [app] Ending request: urn:request:868ea2a6-c55e-4cac-b3ba-2939059496f7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:868ea2a6-c55e-4cac-b3ba-2939059496f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:30,639 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:43:30,639 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.002)
gunicorn-web stdout | 2024-04-22 03:43:30,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:30,641 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:30,642 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:30,642 [286] [DEBUG] [app] Starting request: urn:request:8775a18f-5f7e-4a03-8102-bf9db3ce1807 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:30,643 [286] [DEBUG] [app] Ending request: urn:request:8775a18f-5f7e-4a03-8102-bf9db3ce1807 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8775a18f-5f7e-4a03-8102-bf9db3ce1807', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:30,643 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:43:30,643 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:30,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:30,645 [285] [DEBUG] [app] Starting request: urn:request:b883b0b9-7816-417d-adcb-0f4c716ce67e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:30,645 [285] [DEBUG] [app] Ending request: urn:request:b883b0b9-7816-417d-adcb-0f4c716ce67e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b883b0b9-7816-417d-adcb-0f4c716ce67e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.003)
gunicorn-web stdout | 2024-04-22 03:43:30,646 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:30,646 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:30,647 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:30,654 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:30,656 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:30,667 [282] [DEBUG] [app] Ending request: urn:request:16ee3347-dfe2-47e7-8e5f-7628a30b024d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16ee3347-dfe2-47e7-8e5f-7628a30b024d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:30,667 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:30,667 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.035)
gunicorn-web stdout | 2024-04-22 03:43:30,669 [284] [DEBUG] [app] Ending request: urn:request:3c6e5968-b013-49bf-a687-61d3f2e7c557 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3c6e5968-b013-49bf-a687-61d3f2e7c557', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:30,669 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:30,670 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.038 118 0.038)
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:31.933721+00:00 (in 0.999776 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:48:30 UTC)" (scheduled at 2024-04-22 03:43:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 43, 30, 934207), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,948 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 43, 30, 934207), True, datetime.datetime(2024, 4, 22, 3, 43, 30, 934207), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,953 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 43, 30, 934207), True, datetime.datetime(2024, 4, 22, 3, 43, 30, 934207), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 3, 43, 30, 934207), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,957 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:43:30,957 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:48:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:43:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:43:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:01.842666+00:00 (in 29.999515 seconds)
gcworker stdout | 2024-04-22 03:43:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:01 UTC)" (scheduled at 2024-04-22 03:43:31.842666+00:00)
gcworker stdout | 2024-04-22 03:43:31,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:43:31,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:43:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:43:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:29.931002+00:00 (in 57.996732 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:43:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:31 UTC)" (scheduled at 2024-04-22 03:43:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:43:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:43:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:43:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:42.430973+00:00 (in 9.999475 seconds)
notificationworker stdout | 2024-04-22 03:43:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:42 UTC)" (scheduled at 2024-04-22 03:43:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:43:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:43:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 32, 431767), True, datetime.datetime(2024, 4, 22, 3, 43, 32, 431767), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:43:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:43:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:43:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:43:33,805 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:43:35,039 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:43:37,686 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:43:38,905 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:43:39,209 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:43:39,229 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:43:39,942 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:43:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:43:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:10.238173+00:00 (in 29.999520 seconds)
autopruneworker stdout | 2024-04-22 03:43:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:10 UTC)" (scheduled at 2024-04-22 03:43:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:43:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757360247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:43:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:43:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:43:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:43:40,450 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:43:40,604 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:43:42,003 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:43:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:52.430973+00:00 (in 9.999482 seconds)
notificationworker stdout | 2024-04-22 03:43:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:52 UTC)" (scheduled at 2024-04-22 03:43:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:43:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:43:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 42, 431793), True, datetime.datetime(2024, 4, 22, 3, 43, 42, 431793), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:43:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:43:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:43:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:43:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:43:42,841 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:43:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:43:44,068 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:47.065745+00:00 (in 2.997209 seconds)
securityworker stdout | 2024-04-22 03:43:44,068 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:14 UTC)" (scheduled at 2024-04-22 03:43:44.067050+00:00)
securityworker stdout | 2024-04-22 03:43:44,069 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:44,069 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:43:44,073 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:43:44,073 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:43:44,074 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:44,074 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:43:44,074 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:43:44,080 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:43:44,083 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:43:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:43:44,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:43:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:43:44,108 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 38, 44, 83012), 1, 2])
securityworker stdout | 2024-04-22 03:43:44,112 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:43:44,112 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:43:44,114 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 38, 44, 83012), 1, 2])
securityworker stdout | 2024-04-22 03:43:44,118 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:43:44,118 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:43:44,118 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:44,118 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:43:44,118 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:43:44,118 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:43:44,118 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:43:44,120 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:43:44,120 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:43:44,120 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:43:45,315 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:43:45,610 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:43:45,634 [286] [DEBUG] [app] Starting request: urn:request:54423207-72e0-41b3-9f33-ca9493d49af8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:45,634 [282] [DEBUG] [app] Starting request: urn:request:f74bab67-2946-466e-ac5c-ba803b88c0f3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:43:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:43:45,640 [280] [DEBUG] [app] Starting request: urn:request:0c6fe28d-3842-49bd-984f-cdf0474ae695 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:45,640 [275] [DEBUG] [app] Starting request: urn:request:89447964-3482-4094-ab95-a59a58b82d3f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:43:45,641 [280] [DEBUG] [app] Ending request: urn:request:0c6fe28d-3842-49bd-984f-cdf0474ae695 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c6fe28d-3842-49bd-984f-cdf0474ae695', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:45,641 [275] [DEBUG] [app] Ending request: urn:request:89447964-3482-4094-ab95-a59a58b82d3f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:89447964-3482-4094-ab95-a59a58b82d3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:43:45,641 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.003)
gunicorn-registry stdout | 2024-04-22 03:43:45,642 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:45,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:45,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:45,646 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:45,647 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:43:45,649 [286] [DEBUG] [app] Starting request: urn:request:af6b4e3f-52af-4438-af7c-5d32a9676d6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:45,649 [285] [DEBUG] [app] Starting request: urn:request:d075fa54-b473-4142-a471-2bfab9ac5ef3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:43:45,649 [286] [DEBUG] [app] Ending request: urn:request:af6b4e3f-52af-4438-af7c-5d32a9676d6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af6b4e3f-52af-4438-af7c-5d32a9676d6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:45,650 [285] [DEBUG] [app] Ending request: urn:request:d075fa54-b473-4142-a471-2bfab9ac5ef3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d075fa54-b473-4142-a471-2bfab9ac5ef3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:43:45,650 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:43:45,651 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:43:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.003)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:43:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 159 0.003)
gunicorn-web stdout | 2024-04-22 03:43:45,651 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:45,651 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:43:45,652 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:45,652 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:43:45,662 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:45,662 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:43:45,677 [282] [DEBUG] [app] Ending request: urn:request:f74bab67-2946-466e-ac5c-ba803b88c0f3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f74bab67-2946-466e-ac5c-ba803b88c0f3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:45,677 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:43:45,678 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:43:45,678 [286] [DEBUG] [app] Ending request: urn:request:54423207-72e0-41b3-9f33-ca9493d49af8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54423207-72e0-41b3-9f33-ca9493d49af8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:43:45,679 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.046 118 0.046)
gunicorn-web stdout | 2024-04-22 03:43:45,680 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:43:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:43:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.048 118 0.048)
globalpromstats stdout | 2024-04-22 03:43:46,014 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:43:46,772 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:43:47,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:43:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:14.067050+00:00 (in 27.000413 seconds)
securityworker stdout | 2024-04-22 03:43:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:17 UTC)" (scheduled at 2024-04-22 03:43:47.065745+00:00)
securityworker stdout | 2024-04-22 03:43:47,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:43:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:43:47,071 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:43:47,074 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:43:47,088 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:43:47,092 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:43:47,092 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:43:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:43:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:10.248668+00:00 (in 23.002102 seconds)
namespacegcworker stdout | 2024-04-22 03:43:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:47 UTC)" (scheduled at 2024-04-22 03:43:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:43:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:43:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 47, 246933), True, datetime.datetime(2024, 4, 22, 3, 43, 47, 246933), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:43:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:43:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:43:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:43:47,774 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:43:48,675 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:43:49,186 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:43:49,451 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:43:49,457 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:43:49,461 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:43:50,341 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:43:50,748 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:43:50,818 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:43:50,888 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:43:50,958 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:43:50,966 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:43:50,972 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:43:50,978 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:43:50,988 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:43:50,994 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:43:50,998 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:43:51,004 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:43:51,158 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:43:51,162 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:43:51,170 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:43:51,175 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:43:51,180 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:43:51,608 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:43:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:43:54.433683+00:00 (in 2.002168 seconds)
notificationworker stdout | 2024-04-22 03:43:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:02 UTC)" (scheduled at 2024-04-22 03:43:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:43:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:43:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 52, 431879), True, datetime.datetime(2024, 4, 22, 3, 43, 52, 431879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:43:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:43:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:43:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:43:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:43:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:18.047111+00:00 (in 25.000728 seconds)
exportactionlogsworker stdout | 2024-04-22 03:43:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:53 UTC)" (scheduled at 2024-04-22 03:43:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:43:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:43:53,048 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 43, 53, 46876), True, datetime.datetime(2024, 4, 22, 3, 43, 53, 46876), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:43:53,066 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:43:53,066 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:43:53,067 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:43:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:43:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:02.430973+00:00 (in 7.996572 seconds)
notificationworker stdout | 2024-04-22 03:43:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:54 UTC)" (scheduled at 2024-04-22 03:43:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:43:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:43:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:44:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:43:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:43:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:28.130973+00:00 (in 29.999338 seconds)
buildlogsarchiver stdout | 2024-04-22 03:43:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:28 UTC)" (scheduled at 2024-04-22 03:43:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:43:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 43, 58, 131905), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:43:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:43:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:43:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:44:00,632 [286] [DEBUG] [app] Starting request: urn:request:723dcd2a-3cb8-49a6-8b66-08486fe9acfd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:00,632 [282] [DEBUG] [app] Starting request: urn:request:2728b7e0-fe6d-4815-abd2-03c7bd1f7012 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:44:00,636 [283] [DEBUG] [app] Starting request: urn:request:b36f3442-11c9-436e-a121-ac44beb7e06f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:00,636 [281] [DEBUG] [app] Starting request: urn:request:1dde20c6-06a8-418e-853f-ae67bc7a0401 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:00,637 [281] [DEBUG] [app] Ending request: urn:request:1dde20c6-06a8-418e-853f-ae67bc7a0401 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1dde20c6-06a8-418e-853f-ae67bc7a0401', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:00,637 [283] [DEBUG] [app] Ending request: urn:request:b36f3442-11c9-436e-a121-ac44beb7e06f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b36f3442-11c9-436e-a121-ac44beb7e06f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:00,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:44:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:00,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:00,640 [282] [DEBUG] [app] Starting request: urn:request:e0f65d38-716f-408b-ad8b-2d02d23b0c73 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:00,640 [285] [DEBUG] [app] Starting request: urn:request:b4181f8f-f245-49ef-ad55-f67af7268dda (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:00,640 [282] [DEBUG] [app] Ending request: urn:request:e0f65d38-716f-408b-ad8b-2d02d23b0c73 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0f65d38-716f-408b-ad8b-2d02d23b0c73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:00,641 [285] [DEBUG] [app] Ending request: urn:request:b4181f8f-f245-49ef-ad55-f67af7268dda (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4181f8f-f245-49ef-ad55-f67af7268dda', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:00,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:00,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:44:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:44:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:44:00,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:00,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:00,663 [282] [DEBUG] [app] Ending request: urn:request:2728b7e0-fe6d-4815-abd2-03c7bd1f7012 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2728b7e0-fe6d-4815-abd2-03c7bd1f7012', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:44:00,664 [286] [DEBUG] [app] Ending request: urn:request:723dcd2a-3cb8-49a6-8b66-08486fe9acfd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:723dcd2a-3cb8-49a6-8b66-08486fe9acfd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:44:00,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:00,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:00,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:44:00,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gcworker stdout | 2024-04-22 03:44:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:44:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:31.842666+00:00 (in 29.999492 seconds)
gcworker stdout | 2024-04-22 03:44:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:31 UTC)" (scheduled at 2024-04-22 03:44:01.842666+00:00)
gcworker stdout | 2024-04-22 03:44:01,843 [67] [DEBUG] [__main__] No GC policies found
gcworker stdout | 2024-04-22 03:44:01,843 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:44:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:44:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:09.033206+00:00 (in 6.997150 seconds)
repositorygcworker stdout | 2024-04-22 03:44:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:02 UTC)" (scheduled at 2024-04-22 03:44:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:44:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:44:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:44:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:12.430973+00:00 (in 9.999342 seconds)
notificationworker stdout | 2024-04-22 03:44:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:12 UTC)" (scheduled at 2024-04-22 03:44:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:44:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:44:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 2, 432041), True, datetime.datetime(2024, 4, 22, 3, 44, 2, 432041), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:44:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:44:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:44:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:44:03,842 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:44:05,076 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:44:07,721 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:44:08,943 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
repositorygcworker stdout | 2024-04-22 03:44:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:44:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:02.035613+00:00 (in 53.001896 seconds)
repositorygcworker stdout | 2024-04-22 03:44:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:09 UTC)" (scheduled at 2024-04-22 03:44:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:44:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:44:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 9, 33981), True, datetime.datetime(2024, 4, 22, 3, 44, 9, 33981), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:44:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:44:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:44:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:09 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:44:09,246 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:44:09,264 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:44:09,979 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:44:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:44:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:40.238173+00:00 (in 29.999513 seconds)
autopruneworker stdout | 2024-04-22 03:44:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:40 UTC)" (scheduled at 2024-04-22 03:44:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:44:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757390247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:44:10,249 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:44:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:47.245913+00:00 (in 36.996542 seconds)
namespacegcworker stdout | 2024-04-22 03:44:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:10 UTC)" (scheduled at 2024-04-22 03:44:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:44:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:44:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:44:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:44:10,255 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:44:10,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:44:10,487 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:44:10,642 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:44:12,040 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:44:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:22.430973+00:00 (in 9.999516 seconds)
notificationworker stdout | 2024-04-22 03:44:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:22 UTC)" (scheduled at 2024-04-22 03:44:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:44:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:44:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 12, 431900), True, datetime.datetime(2024, 4, 22, 3, 44, 12, 431900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:44:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:44:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:44:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:44:12,878 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:44:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:44:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:17.065745+00:00 (in 2.998148 seconds)
securityworker stdout | 2024-04-22 03:44:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:44 UTC)" (scheduled at 2024-04-22 03:44:14.067050+00:00)
securityworker stdout | 2024-04-22 03:44:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:44:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:44:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:44:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:44:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:44:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:44:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:44:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:44:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:44:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:44:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 39, 14, 77259), 1, 2])
securityworker stdout | 2024-04-22 03:44:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:44:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:44:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 39, 14, 77259), 1, 2])
securityworker stdout | 2024-04-22 03:44:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:44:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:44:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:44:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:44:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:44:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:44:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:44:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:44:15,352 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:44:15,632 [284] [DEBUG] [app] Starting request: urn:request:bbb489f9-610d-4bd7-a69e-8c39d3c4aeaf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:15,632 [286] [DEBUG] [app] Starting request: urn:request:b148cbc1-9c8d-4ae2-a727-fa01166e8d1a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:44:15,636 [280] [DEBUG] [app] Starting request: urn:request:9c478254-7250-41ae-9dae-616ee7a1df67 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:15,636 [281] [DEBUG] [app] Starting request: urn:request:327be467-124d-46f3-9ec6-bc0c68eade82 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:15,636 [280] [DEBUG] [app] Ending request: urn:request:9c478254-7250-41ae-9dae-616ee7a1df67 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c478254-7250-41ae-9dae-616ee7a1df67', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:15,636 [281] [DEBUG] [app] Ending request: urn:request:327be467-124d-46f3-9ec6-bc0c68eade82 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:327be467-124d-46f3-9ec6-bc0c68eade82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:15,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:44:15,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:44:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:15,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:15,639 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:15,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:15,640 [284] [DEBUG] [app] Starting request: urn:request:781032dd-1343-43d7-8023-e054717594c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:15,640 [284] [DEBUG] [app] Ending request: urn:request:781032dd-1343-43d7-8023-e054717594c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:781032dd-1343-43d7-8023-e054717594c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:15,641 [286] [DEBUG] [app] Starting request: urn:request:e3676503-aa17-4a33-b1da-e9da6161da28 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:44:15,641 [286] [DEBUG] [app] Ending request: urn:request:e3676503-aa17-4a33-b1da-e9da6161da28 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e3676503-aa17-4a33-b1da-e9da6161da28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:15,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:15,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:15,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:44:15,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:15,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
queuecleanupworker stdout | 2024-04-22 03:44:15,647 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
gunicorn-web stdout | 2024-04-22 03:44:15,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:15,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:15,662 [284] [DEBUG] [app] Ending request: urn:request:bbb489f9-610d-4bd7-a69e-8c39d3c4aeaf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbb489f9-610d-4bd7-a69e-8c39d3c4aeaf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:44:15,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:15,663 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:44:15,663 [286] [DEBUG] [app] Ending request: urn:request:b148cbc1-9c8d-4ae2-a727-fa01166e8d1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b148cbc1-9c8d-4ae2-a727-fa01166e8d1a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:44:15,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:15,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
globalpromstats stdout | 2024-04-22 03:44:16,052 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:44:16,809 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:44:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:44:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:44.067050+00:00 (in 27.000752 seconds)
securityworker stdout | 2024-04-22 03:44:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:47 UTC)" (scheduled at 2024-04-22 03:44:17.065745+00:00)
securityworker stdout | 2024-04-22 03:44:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:44:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:44:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:44:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:44:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:44:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:44:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:44:17,806 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:44:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:44:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:53.045554+00:00 (in 34.997726 seconds)
exportactionlogsworker stdout | 2024-04-22 03:44:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:18 UTC)" (scheduled at 2024-04-22 03:44:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:44:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:44:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:44:18,714 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:44:19,223 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:44:19,462 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:44:19,466 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:44:19,471 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:44:20,378 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:44:20,777 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:44:20,855 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:44:20,897 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:44:20,969 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:44:20,976 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:44:20,981 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:44:20,987 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:44:20,999 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:44:21,003 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:44:21,007 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:44:21,014 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:44:21,168 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:44:21,171 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:44:21,178 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:44:21,183 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:44:21,187 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:44:21,646 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:44:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:32.430973+00:00 (in 9.999496 seconds)
notificationworker stdout | 2024-04-22 03:44:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:32 UTC)" (scheduled at 2024-04-22 03:44:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:44:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:44:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 22, 431774), True, datetime.datetime(2024, 4, 22, 3, 44, 22, 431774), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:44:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:44:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:44:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:44:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:44:24,739 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:24.738365+00:00 (in 59.999325 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:44:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:24 UTC)" (scheduled at 2024-04-22 03:44:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:44:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:44:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:44:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:44:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:44:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:58.130973+00:00 (in 29.999469 seconds)
buildlogsarchiver stdout | 2024-04-22 03:44:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:58 UTC)" (scheduled at 2024-04-22 03:44:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:44:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 44, 28, 131708), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:44:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:44:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:44:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:44:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:31.933721+00:00 (in 2.002215 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:29 UTC)" (scheduled at 2024-04-22 03:44:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,933 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 29, 931751), True, datetime.datetime(2024, 4, 22, 3, 44, 29, 931751), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:44:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:44:30,632 [286] [DEBUG] [app] Starting request: urn:request:03d9be78-0727-41b7-9121-7f0a1bf4264a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:30,632 [284] [DEBUG] [app] Starting request: urn:request:632feb35-7a8e-44a9-9281-eefbc4e8efb7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:44:30,636 [280] [DEBUG] [app] Starting request: urn:request:5fbb8307-5c71-424b-9fd2-a8184a7dfffd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:30,636 [283] [DEBUG] [app] Starting request: urn:request:d3e14ab1-5f38-49e3-bec4-e2215fd020c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:30,636 [280] [DEBUG] [app] Ending request: urn:request:5fbb8307-5c71-424b-9fd2-a8184a7dfffd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5fbb8307-5c71-424b-9fd2-a8184a7dfffd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:30,636 [283] [DEBUG] [app] Ending request: urn:request:d3e14ab1-5f38-49e3-bec4-e2215fd020c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3e14ab1-5f38-49e3-bec4-e2215fd020c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:30,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:44:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:30,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:30,639 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:30,640 [286] [DEBUG] [app] Starting request: urn:request:898e2672-9b6e-4e88-8048-c59e97e09391 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:30,641 [286] [DEBUG] [app] Ending request: urn:request:898e2672-9b6e-4e88-8048-c59e97e09391 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:898e2672-9b6e-4e88-8048-c59e97e09391', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:30,641 [285] [DEBUG] [app] Starting request: urn:request:a57fdf0a-1894-4d85-a408-b20caf4c6efc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:30,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:44:30,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:30,641 [285] [DEBUG] [app] Ending request: urn:request:a57fdf0a-1894-4d85-a408-b20caf4c6efc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a57fdf0a-1894-4d85-a408-b20caf4c6efc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:30,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:44:30,642 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:30,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:44:30,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:44:30,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:30,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:30,662 [284] [DEBUG] [app] Ending request: urn:request:632feb35-7a8e-44a9-9281-eefbc4e8efb7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:632feb35-7a8e-44a9-9281-eefbc4e8efb7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:44:30,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:30,663 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:44:30,663 [286] [DEBUG] [app] Ending request: urn:request:03d9be78-0727-41b7-9121-7f0a1bf4264a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:03d9be78-0727-41b7-9121-7f0a1bf4264a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:44:30,663 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.033)
gunicorn-web stdout | 2024-04-22 03:44:30,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:44:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:44:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:01.842666+00:00 (in 29.999388 seconds)
gcworker stdout | 2024-04-22 03:44:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:01 UTC)" (scheduled at 2024-04-22 03:44:31.842666+00:00)
gcworker stdout | 2024-04-22 03:44:31,843 [67] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2024-04-22 03:44:31,858 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712547871857, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:44:31,866 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:44:31,866 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:44:31,866 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:44:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:44:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:29.931002+00:00 (in 57.996833 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:44:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:31 UTC)" (scheduled at 2024-04-22 03:44:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:44:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:44:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:44:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:42.430973+00:00 (in 9.999286 seconds)
notificationworker stdout | 2024-04-22 03:44:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:42 UTC)" (scheduled at 2024-04-22 03:44:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:44:32,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:44:32,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 32, 432250), True, datetime.datetime(2024, 4, 22, 3, 44, 32, 432250), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:44:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:44:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:44:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:44:33,879 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:44:35,113 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:44:37,759 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:44:38,981 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:44:39,283 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:44:39,301 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:44:40,017 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:44:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:44:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:10.238173+00:00 (in 29.999513 seconds)
autopruneworker stdout | 2024-04-22 03:44:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:10 UTC)" (scheduled at 2024-04-22 03:44:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:44:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757420247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:44:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:44:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:44:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:44:40,525 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:44:40,678 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:44:42,079 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:44:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:52.430973+00:00 (in 9.999412 seconds)
notificationworker stdout | 2024-04-22 03:44:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:52 UTC)" (scheduled at 2024-04-22 03:44:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:44:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:44:42,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 42, 431958), True, datetime.datetime(2024, 4, 22, 3, 44, 42, 431958), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:44:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:44:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:44:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:44:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:44:42,915 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:44:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:44:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:47.065745+00:00 (in 2.998052 seconds)
securityworker stdout | 2024-04-22 03:44:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:14 UTC)" (scheduled at 2024-04-22 03:44:44.067050+00:00)
securityworker stdout | 2024-04-22 03:44:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:44:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:44:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:44:44,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:44:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:44:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:44:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:44:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:44:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:44:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:44:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 39, 44, 77921), 1, 2])
securityworker stdout | 2024-04-22 03:44:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:44:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:44:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 39, 44, 77921), 1, 2])
securityworker stdout | 2024-04-22 03:44:44,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:44:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:44:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:44:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:44:44,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:44:44,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:44:44,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:44:44,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:44:44,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:44:45,369 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:44:45,632 [285] [DEBUG] [app] Starting request: urn:request:602cc1a4-1621-4b3e-a74c-1f53b26b167a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:45,632 [282] [DEBUG] [app] Starting request: urn:request:6c0b5e79-a5b7-4b8a-9450-dec1551fdcd3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:44:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:44:45,636 [275] [DEBUG] [app] Starting request: urn:request:38bf19d3-404b-45b8-aeb8-8a43492e2264 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:45,636 [279] [DEBUG] [app] Starting request: urn:request:233ce3c9-d2a7-4b62-8ad6-e141e900ed74 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:44:45,636 [275] [DEBUG] [app] Ending request: urn:request:38bf19d3-404b-45b8-aeb8-8a43492e2264 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:38bf19d3-404b-45b8-aeb8-8a43492e2264', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:45,636 [279] [DEBUG] [app] Ending request: urn:request:233ce3c9-d2a7-4b62-8ad6-e141e900ed74 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:233ce3c9-d2a7-4b62-8ad6-e141e900ed74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:44:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:44:45,637 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:44:45,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:45,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:44:45,640 [282] [DEBUG] [app] Starting request: urn:request:c0823454-7a40-4cbc-b371-7f871cd75c2d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:44:45,640 [282] [DEBUG] [app] Ending request: urn:request:c0823454-7a40-4cbc-b371-7f871cd75c2d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c0823454-7a40-4cbc-b371-7f871cd75c2d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:45,640 [286] [DEBUG] [app] Starting request: urn:request:36669ec9-0e45-44cf-98d8-1466e0604bd1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:44:45,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:44:45,641 [286] [DEBUG] [app] Ending request: urn:request:36669ec9-0e45-44cf-98d8-1466e0604bd1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:36669ec9-0e45-44cf-98d8-1466e0604bd1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:44:45,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:45,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:44:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:44:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:44:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:44:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:44:45,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:44:45,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:45,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:44:45,663 [282] [DEBUG] [app] Ending request: urn:request:6c0b5e79-a5b7-4b8a-9450-dec1551fdcd3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6c0b5e79-a5b7-4b8a-9450-dec1551fdcd3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:44:45,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:45,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:44:45,664 [285] [DEBUG] [app] Ending request: urn:request:602cc1a4-1621-4b3e-a74c-1f53b26b167a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:602cc1a4-1621-4b3e-a74c-1f53b26b167a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:44:45,664 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:44:45,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:44:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:44:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.034)
queuecleanupworker stdout | 2024-04-22 03:44:45,684 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:44:46,060 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:44:46,847 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:44:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:44:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:14.067050+00:00 (in 27.000707 seconds)
securityworker stdout | 2024-04-22 03:44:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:17 UTC)" (scheduled at 2024-04-22 03:44:47.065745+00:00)
securityworker stdout | 2024-04-22 03:44:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:44:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:44:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:44:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:44:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:44:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:44:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:44:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:44:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:10.248668+00:00 (in 23.002117 seconds)
namespacegcworker stdout | 2024-04-22 03:44:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:47 UTC)" (scheduled at 2024-04-22 03:44:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:44:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:44:47,248 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 47, 247006), True, datetime.datetime(2024, 4, 22, 3, 44, 47, 247006), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:44:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:44:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:44:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:44:47,843 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:44:48,751 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:44:49,260 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:44:49,473 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:44:49,476 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:44:49,480 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:44:50,415 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:44:50,814 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:44:50,891 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:44:50,906 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:44:50,981 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:44:50,986 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:44:50,989 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:44:50,996 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:44:51,008 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:44:51,014 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:44:51,017 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:44:51,022 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:44:51,179 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:44:51,182 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:44:51,185 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:44:51,190 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:44:51,195 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:44:51,683 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:44:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:44:54.433683+00:00 (in 2.002170 seconds)
notificationworker stdout | 2024-04-22 03:44:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:02 UTC)" (scheduled at 2024-04-22 03:44:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:44:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:44:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 52, 432007), True, datetime.datetime(2024, 4, 22, 3, 44, 52, 432007), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:44:52,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:44:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:44:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:44:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:44:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:18.047111+00:00 (in 25.000921 seconds)
exportactionlogsworker stdout | 2024-04-22 03:44:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:53 UTC)" (scheduled at 2024-04-22 03:44:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:44:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:44:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 44, 53, 46587), True, datetime.datetime(2024, 4, 22, 3, 44, 53, 46587), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:44:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:44:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:44:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:44:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:44:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:02.430973+00:00 (in 7.996484 seconds)
notificationworker stdout | 2024-04-22 03:44:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:54 UTC)" (scheduled at 2024-04-22 03:44:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:44:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:44:54,435 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:45:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:44:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:44:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:28.130973+00:00 (in 29.999489 seconds)
buildlogsarchiver stdout | 2024-04-22 03:44:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:28 UTC)" (scheduled at 2024-04-22 03:44:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:44:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 44, 58, 131736), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:44:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:44:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:44:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:45:00,632 [282] [DEBUG] [app] Starting request: urn:request:1937535a-0975-43a8-94f4-45e134a5a96a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:00,632 [284] [DEBUG] [app] Starting request: urn:request:ccfa8b54-f271-4dfe-913e-42288778682b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:45:00,635 [275] [DEBUG] [app] Starting request: urn:request:25f79a29-a382-426c-881c-68472f01ede3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:00,636 [275] [DEBUG] [app] Ending request: urn:request:25f79a29-a382-426c-881c-68472f01ede3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:25f79a29-a382-426c-881c-68472f01ede3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:45:00,636 [279] [DEBUG] [app] Starting request: urn:request:7441ab93-fb71-49c2-ae23-8eb7d6aa0b97 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:45:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:45:00,636 [279] [DEBUG] [app] Ending request: urn:request:7441ab93-fb71-49c2-ae23-8eb7d6aa0b97 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7441ab93-fb71-49c2-ae23-8eb7d6aa0b97', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:45:00,637 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:45:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:00,640 [286] [DEBUG] [app] Starting request: urn:request:83688afc-bf76-4ee9-8f23-ab97130e508f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:00,640 [285] [DEBUG] [app] Starting request: urn:request:d12bf781-747e-4997-a65a-8f8b4fe6a347 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:00,641 [286] [DEBUG] [app] Ending request: urn:request:83688afc-bf76-4ee9-8f23-ab97130e508f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83688afc-bf76-4ee9-8f23-ab97130e508f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:00,641 [285] [DEBUG] [app] Ending request: urn:request:d12bf781-747e-4997-a65a-8f8b4fe6a347 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d12bf781-747e-4997-a65a-8f8b4fe6a347', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:45:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:00,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:45:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:00,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:00,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:00,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:00,663 [282] [DEBUG] [app] Ending request: urn:request:1937535a-0975-43a8-94f4-45e134a5a96a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1937535a-0975-43a8-94f4-45e134a5a96a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:00,663 [284] [DEBUG] [app] Ending request: urn:request:ccfa8b54-f271-4dfe-913e-42288778682b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ccfa8b54-f271-4dfe-913e-42288778682b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:00,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:00,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:00,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:45:00,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.033)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gcworker stdout | 2024-04-22 03:45:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:45:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:31.842666+00:00 (in 29.999416 seconds)
gcworker stdout | 2024-04-22 03:45:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:31 UTC)" (scheduled at 2024-04-22 03:45:01.842666+00:00)
gcworker stdout | 2024-04-22 03:45:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712547901843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:45:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:45:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:45:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:45:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:45:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:09.033206+00:00 (in 6.997011 seconds)
repositorygcworker stdout | 2024-04-22 03:45:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:02 UTC)" (scheduled at 2024-04-22 03:45:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:45:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:45:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:45:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:12.430973+00:00 (in 9.999305 seconds)
notificationworker stdout | 2024-04-22 03:45:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:12 UTC)" (scheduled at 2024-04-22 03:45:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:45:02,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:45:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 2, 432176), True, datetime.datetime(2024, 4, 22, 3, 45, 2, 432176), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:45:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:45:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:45:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:45:03,916 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:45:05,150 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:45:07,797 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:45:08,990 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
repositorygcworker stdout | 2024-04-22 03:45:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:45:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:02.035613+00:00 (in 53.001966 seconds)
repositorygcworker stdout | 2024-04-22 03:45:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:09 UTC)" (scheduled at 2024-04-22 03:45:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:45:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:45:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 9, 34121), True, datetime.datetime(2024, 4, 22, 3, 45, 9, 34121), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:45:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:45:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:45:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:09 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:45:09,321 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:45:09,337 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:45:10,054 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:45:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:45:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:40.238173+00:00 (in 29.999509 seconds)
autopruneworker stdout | 2024-04-22 03:45:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:40 UTC)" (scheduled at 2024-04-22 03:45:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:45:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757450247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:45:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:45:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:47.245913+00:00 (in 36.996697 seconds)
namespacegcworker stdout | 2024-04-22 03:45:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:10 UTC)" (scheduled at 2024-04-22 03:45:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:45:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:45:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:45:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:45:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:45:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:45:10,562 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:45:10,715 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:45:12,116 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:45:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:22.430973+00:00 (in 9.999407 seconds)
notificationworker stdout | 2024-04-22 03:45:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:22 UTC)" (scheduled at 2024-04-22 03:45:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:45:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:45:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 12, 431965), True, datetime.datetime(2024, 4, 22, 3, 45, 12, 431965), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:45:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:45:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:45:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:45:12,952 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:45:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:45:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:17.065745+00:00 (in 2.998090 seconds)
securityworker stdout | 2024-04-22 03:45:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:44 UTC)" (scheduled at 2024-04-22 03:45:14.067050+00:00)
securityworker stdout | 2024-04-22 03:45:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:45:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:45:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:45:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:45:14,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:45:14,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:45:14,080 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:45:14,093 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:45:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:45:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:45:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 40, 14, 79702), 1, 2])
securityworker stdout | 2024-04-22 03:45:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:45:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:45:14,104 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 40, 14, 79702), 1, 2])
securityworker stdout | 2024-04-22 03:45:14,108 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:45:14,108 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:45:14,108 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:14,108 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:45:14,108 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:45:14,108 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:45:14,108 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:45:14,109 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:14,109 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:45:14,110 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:45:15,406 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:45:15,631 [284] [DEBUG] [app] Starting request: urn:request:16fe2910-16d7-4e1f-aee9-2d8fc18c3b4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:15,631 [282] [DEBUG] [app] Starting request: urn:request:8cf30e49-ab29-4219-9b2a-d2f933024696 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:45:15,635 [281] [DEBUG] [app] Starting request: urn:request:ffda34c0-3d22-44da-864a-a6ff8a6c63b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:15,635 [275] [DEBUG] [app] Starting request: urn:request:2c83e836-c955-4e90-9f91-aade0864a655 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:15,635 [281] [DEBUG] [app] Ending request: urn:request:ffda34c0-3d22-44da-864a-a6ff8a6c63b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ffda34c0-3d22-44da-864a-a6ff8a6c63b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:45:15,635 [275] [DEBUG] [app] Ending request: urn:request:2c83e836-c955-4e90-9f91-aade0864a655 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c83e836-c955-4e90-9f91-aade0864a655', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:45:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:45:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:45:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:15,639 [284] [DEBUG] [app] Starting request: urn:request:691d6bac-f845-45a8-9001-7aa0883a5673 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:15,639 [284] [DEBUG] [app] Ending request: urn:request:691d6bac-f845-45a8-9001-7aa0883a5673 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:691d6bac-f845-45a8-9001-7aa0883a5673', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:15,639 [286] [DEBUG] [app] Starting request: urn:request:563118e4-3614-4a15-9f20-4b481f878700 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:15,640 [286] [DEBUG] [app] Ending request: urn:request:563118e4-3614-4a15-9f20-4b481f878700 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:563118e4-3614-4a15-9f20-4b481f878700', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:45:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:15,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:15,662 [284] [DEBUG] [app] Ending request: urn:request:16fe2910-16d7-4e1f-aee9-2d8fc18c3b4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:16fe2910-16d7-4e1f-aee9-2d8fc18c3b4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:15,662 [282] [DEBUG] [app] Ending request: urn:request:8cf30e49-ab29-4219-9b2a-d2f933024696 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8cf30e49-ab29-4219-9b2a-d2f933024696', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:15,662 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:15,662 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:45:15,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
queuecleanupworker stdout | 2024-04-22 03:45:15,721 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:45:16,097 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:45:16,887 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:45:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:45:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:44.067050+00:00 (in 27.000805 seconds)
securityworker stdout | 2024-04-22 03:45:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:47 UTC)" (scheduled at 2024-04-22 03:45:17.065745+00:00)
securityworker stdout | 2024-04-22 03:45:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:45:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:45:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:45:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:45:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:45:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:45:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:45:17,880 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:45:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:45:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:53.045554+00:00 (in 34.997774 seconds)
exportactionlogsworker stdout | 2024-04-22 03:45:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:18 UTC)" (scheduled at 2024-04-22 03:45:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:45:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:45:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:45:18,788 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:45:19,297 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:45:19,482 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:45:19,487 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:45:19,490 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:45:20,454 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:45:20,850 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:45:20,915 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
builder stdout | 2024-04-22 03:45:20,929 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:45:20,999 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:45:21,002 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:45:21,006 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:45:21,009 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:45:21,016 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:45:21,025 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:45:21,030 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:45:21,034 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:45:21,190 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:45:21,194 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:45:21,198 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:45:21,203 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:45:21,206 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:45:21,722 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:45:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:32.430973+00:00 (in 9.999393 seconds)
notificationworker stdout | 2024-04-22 03:45:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:32 UTC)" (scheduled at 2024-04-22 03:45:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:45:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:45:22,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 22, 432109), True, datetime.datetime(2024, 4, 22, 3, 45, 22, 432109), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:45:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:45:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:45:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:45:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:45:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:24.738365+00:00 (in 59.999440 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:45:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:24 UTC)" (scheduled at 2024-04-22 03:45:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:45:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:45:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:45:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:45:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:45:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:58.130973+00:00 (in 29.999444 seconds)
buildlogsarchiver stdout | 2024-04-22 03:45:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:58 UTC)" (scheduled at 2024-04-22 03:45:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:45:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 45, 28, 131928), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:45:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:45:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:45:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:45:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:31.933721+00:00 (in 2.002179 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:29 UTC)" (scheduled at 2024-04-22 03:45:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 29, 931829), True, datetime.datetime(2024, 4, 22, 3, 45, 29, 931829), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:45:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:45:30,632 [286] [DEBUG] [app] Starting request: urn:request:483f5006-9616-4904-9c88-cf64afc263dd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:30,632 [285] [DEBUG] [app] Starting request: urn:request:3cdbd0c9-edcd-4cce-86a6-e6209e4a8873 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:45:30,636 [275] [DEBUG] [app] Starting request: urn:request:af151c4b-489b-413a-baba-4e9312162e2b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:30,636 [283] [DEBUG] [app] Starting request: urn:request:e3a5cf16-fb77-49ce-a0bc-43ae45f56375 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:30,637 [275] [DEBUG] [app] Ending request: urn:request:af151c4b-489b-413a-baba-4e9312162e2b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:af151c4b-489b-413a-baba-4e9312162e2b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:45:30,637 [283] [DEBUG] [app] Ending request: urn:request:e3a5cf16-fb77-49ce-a0bc-43ae45f56375 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e3a5cf16-fb77-49ce-a0bc-43ae45f56375', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:45:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:45:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:30,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:45:30,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:30,641 [284] [DEBUG] [app] Starting request: urn:request:dda27966-fdd2-4e31-aaef-29a150b8eb07 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:30,641 [282] [DEBUG] [app] Starting request: urn:request:485fd069-bff7-4b69-a654-688ff2466fdd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:30,641 [284] [DEBUG] [app] Ending request: urn:request:dda27966-fdd2-4e31-aaef-29a150b8eb07 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dda27966-fdd2-4e31-aaef-29a150b8eb07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:45:30,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:30,642 [282] [DEBUG] [app] Ending request: urn:request:485fd069-bff7-4b69-a654-688ff2466fdd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:485fd069-bff7-4b69-a654-688ff2466fdd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:30,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:30,642 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:30,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:45:30,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:30,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:30,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:30,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:30,663 [286] [DEBUG] [app] Ending request: urn:request:483f5006-9616-4904-9c88-cf64afc263dd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:483f5006-9616-4904-9c88-cf64afc263dd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:30,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:30,664 [285] [DEBUG] [app] Ending request: urn:request:3cdbd0c9-edcd-4cce-86a6-e6209e4a8873 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3cdbd0c9-edcd-4cce-86a6-e6209e4a8873', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:30,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:45:30,664 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:45:30,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gcworker stdout | 2024-04-22 03:45:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:45:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:01.842666+00:00 (in 29.999318 seconds)
gcworker stdout | 2024-04-22 03:45:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:01 UTC)" (scheduled at 2024-04-22 03:45:31.842666+00:00)
gcworker stdout | 2024-04-22 03:45:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712547931843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:45:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:45:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:45:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:45:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:45:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:29.931002+00:00 (in 57.996702 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:45:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:31 UTC)" (scheduled at 2024-04-22 03:45:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:45:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:45:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:45:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:42.430973+00:00 (in 9.999423 seconds)
notificationworker stdout | 2024-04-22 03:45:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:42 UTC)" (scheduled at 2024-04-22 03:45:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:45:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:45:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 32, 431806), True, datetime.datetime(2024, 4, 22, 3, 45, 32, 431806), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:45:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:45:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:45:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:45:33,954 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:45:35,187 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:45:37,831 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:45:39,027 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:45:39,350 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:45:39,373 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:45:40,075 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:45:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:45:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:10.238173+00:00 (in 29.999492 seconds)
autopruneworker stdout | 2024-04-22 03:45:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:10 UTC)" (scheduled at 2024-04-22 03:45:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:45:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757480247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:45:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:45:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:45:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:45:40,599 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:45:40,751 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:45:42,154 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:45:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:52.430973+00:00 (in 9.999530 seconds)
notificationworker stdout | 2024-04-22 03:45:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:52 UTC)" (scheduled at 2024-04-22 03:45:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:45:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:45:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 42, 431719), True, datetime.datetime(2024, 4, 22, 3, 45, 42, 431719), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:45:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:45:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:45:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:45:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:45:42,989 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:45:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:45:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:47.065745+00:00 (in 2.998085 seconds)
securityworker stdout | 2024-04-22 03:45:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:14 UTC)" (scheduled at 2024-04-22 03:45:44.067050+00:00)
securityworker stdout | 2024-04-22 03:45:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:45:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:45:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:45:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:45:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:45:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:45:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:45:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:45:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:45:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:45:44,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 40, 44, 78332), 1, 2])
securityworker stdout | 2024-04-22 03:45:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:45:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:45:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 40, 44, 78332), 1, 2])
securityworker stdout | 2024-04-22 03:45:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:45:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:45:44,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:44,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:45:44,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:45:44,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:45:44,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:45:44,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:45:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:45:44,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:45:45,443 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:45:45,632 [282] [DEBUG] [app] Starting request: urn:request:77729ca0-82a2-4256-8e90-7d6cdd9197a1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:45,632 [286] [DEBUG] [app] Starting request: urn:request:dfd49434-73ba-45ed-9b62-11b838ff1c8b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:45:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:45,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:45:45,636 [280] [DEBUG] [app] Starting request: urn:request:c760ccd3-1d16-4055-b34e-ee79c72d9d3a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:45,637 [277] [DEBUG] [app] Starting request: urn:request:f94a05a5-3fff-431f-85ee-2a6c27e620cc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:45:45,637 [280] [DEBUG] [app] Ending request: urn:request:c760ccd3-1d16-4055-b34e-ee79c72d9d3a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c760ccd3-1d16-4055-b34e-ee79c72d9d3a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:45:45,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:45:45,637 [277] [DEBUG] [app] Ending request: urn:request:f94a05a5-3fff-431f-85ee-2a6c27e620cc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f94a05a5-3fff-431f-85ee-2a6c27e620cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:45:45,638 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:45:45,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:45,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:45,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:45:45,641 [282] [DEBUG] [app] Starting request: urn:request:4d693901-ed96-45ba-889d-31d0da44bef6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:45,641 [284] [DEBUG] [app] Starting request: urn:request:9be7907b-2a3a-4c2c-93b8-ac5e565a992d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:45:45,641 [282] [DEBUG] [app] Ending request: urn:request:4d693901-ed96-45ba-889d-31d0da44bef6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d693901-ed96-45ba-889d-31d0da44bef6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:45,641 [284] [DEBUG] [app] Ending request: urn:request:9be7907b-2a3a-4c2c-93b8-ac5e565a992d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9be7907b-2a3a-4c2c-93b8-ac5e565a992d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:45:45,642 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:45:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:45:45,642 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:45:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:45:45,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:45,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:45:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:45,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:45:45,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:45,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:45:45,663 [286] [DEBUG] [app] Ending request: urn:request:dfd49434-73ba-45ed-9b62-11b838ff1c8b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dfd49434-73ba-45ed-9b62-11b838ff1c8b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:45,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:45,663 [282] [DEBUG] [app] Ending request: urn:request:77729ca0-82a2-4256-8e90-7d6cdd9197a1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:77729ca0-82a2-4256-8e90-7d6cdd9197a1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:45:45,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:45:45,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:45:45,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:45:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:45:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
queuecleanupworker stdout | 2024-04-22 03:45:45,758 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:45:46,134 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:45:46,924 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:45:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:45:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:14.067050+00:00 (in 27.000733 seconds)
securityworker stdout | 2024-04-22 03:45:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:17 UTC)" (scheduled at 2024-04-22 03:45:47.065745+00:00)
securityworker stdout | 2024-04-22 03:45:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:45:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:45:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:45:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:45:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:45:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:45:47,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:45:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:45:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:10.248668+00:00 (in 23.002260 seconds)
namespacegcworker stdout | 2024-04-22 03:45:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:47 UTC)" (scheduled at 2024-04-22 03:45:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:45:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:45:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 47, 246838), True, datetime.datetime(2024, 4, 22, 3, 45, 47, 246838), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:45:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:45:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:45:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:45:47,917 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:45:48,825 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:45:49,335 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:45:49,492 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:45:49,494 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:45:49,498 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:45:50,491 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:45:50,887 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:45:50,928 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
builder stdout | 2024-04-22 03:45:50,968 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:45:51,009 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:45:51,013 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:45:51,017 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:45:51,020 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:45:51,025 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:45:51,032 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:45:51,037 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:45:51,042 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:45:51,200 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:45:51,204 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:45:51,206 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:45:51,210 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:45:51,213 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:45:51,759 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:45:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:45:54.433683+00:00 (in 2.002169 seconds)
notificationworker stdout | 2024-04-22 03:45:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:02 UTC)" (scheduled at 2024-04-22 03:45:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:45:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:45:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 52, 431828), True, datetime.datetime(2024, 4, 22, 3, 45, 52, 431828), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:45:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:45:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:45:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:45:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:45:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:18.047111+00:00 (in 25.000952 seconds)
exportactionlogsworker stdout | 2024-04-22 03:45:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:53 UTC)" (scheduled at 2024-04-22 03:45:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:45:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:45:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 45, 53, 46431), True, datetime.datetime(2024, 4, 22, 3, 45, 53, 46431), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:45:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:45:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:45:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:45:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:45:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:02.430973+00:00 (in 7.996724 seconds)
notificationworker stdout | 2024-04-22 03:45:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:54 UTC)" (scheduled at 2024-04-22 03:45:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:45:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:45:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:46:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:45:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:45:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:28.130973+00:00 (in 29.999441 seconds)
buildlogsarchiver stdout | 2024-04-22 03:45:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:28 UTC)" (scheduled at 2024-04-22 03:45:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:45:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 45, 58, 131794), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:45:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:45:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:45:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:46:00,632 [282] [DEBUG] [app] Starting request: urn:request:17c0badd-1367-4c88-90b7-a1935baa8a13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:00,632 [284] [DEBUG] [app] Starting request: urn:request:d48ed21d-4fb0-4772-979c-bf01a8be8ce0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:46:00,636 [283] [DEBUG] [app] Starting request: urn:request:3ef5d08b-11d2-491f-9986-6ed4b4f7da2c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:00,636 [276] [DEBUG] [app] Starting request: urn:request:3c2f954f-b1d6-4a01-94c8-c0ef07fc892c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:00,636 [283] [DEBUG] [app] Ending request: urn:request:3ef5d08b-11d2-491f-9986-6ed4b4f7da2c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3ef5d08b-11d2-491f-9986-6ed4b4f7da2c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:00,636 [276] [DEBUG] [app] Ending request: urn:request:3c2f954f-b1d6-4a01-94c8-c0ef07fc892c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c2f954f-b1d6-4a01-94c8-c0ef07fc892c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:46:00,637 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:46:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:46:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:00,640 [286] [DEBUG] [app] Starting request: urn:request:60e6dda8-b7cd-4d6a-83a3-fea6e2ff3de9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:00,640 [286] [DEBUG] [app] Ending request: urn:request:60e6dda8-b7cd-4d6a-83a3-fea6e2ff3de9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:60e6dda8-b7cd-4d6a-83a3-fea6e2ff3de9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:00,640 [282] [DEBUG] [app] Starting request: urn:request:6aa9b3f8-fe00-47bc-ad66-0638a1618c78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:00,641 [282] [DEBUG] [app] Ending request: urn:request:6aa9b3f8-fe00-47bc-ad66-0638a1618c78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6aa9b3f8-fe00-47bc-ad66-0638a1618c78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:46:00,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:46:00,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:46:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:00,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:00,652 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:00,662 [282] [DEBUG] [app] Ending request: urn:request:17c0badd-1367-4c88-90b7-a1935baa8a13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:17c0badd-1367-4c88-90b7-a1935baa8a13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:00,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:46:00,663 [284] [DEBUG] [app] Ending request: urn:request:d48ed21d-4fb0-4772-979c-bf01a8be8ce0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d48ed21d-4fb0-4772-979c-bf01a8be8ce0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:00,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:00,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gcworker stdout | 2024-04-22 03:46:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:46:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:31.842666+00:00 (in 29.999412 seconds)
gcworker stdout | 2024-04-22 03:46:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:31 UTC)" (scheduled at 2024-04-22 03:46:01.842666+00:00)
gcworker stdout | 2024-04-22 03:46:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712547961843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:46:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:46:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:46:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:46:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:46:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:09.033206+00:00 (in 6.997032 seconds)
repositorygcworker stdout | 2024-04-22 03:46:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:02 UTC)" (scheduled at 2024-04-22 03:46:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:46:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:46:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:46:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:12.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 03:46:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:12 UTC)" (scheduled at 2024-04-22 03:46:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:46:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:46:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 2, 431737), True, datetime.datetime(2024, 4, 22, 3, 46, 2, 431737), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:46:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:46:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:46:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:46:03,992 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:46:05,224 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:46:07,868 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:46:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:46:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:02.035613+00:00 (in 53.001928 seconds)
repositorygcworker stdout | 2024-04-22 03:46:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:09 UTC)" (scheduled at 2024-04-22 03:46:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:46:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:46:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 9, 34175), True, datetime.datetime(2024, 4, 22, 3, 46, 9, 34175), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:46:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:46:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:46:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:46:09,064 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:46:09,372 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:46:09,412 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:46:10,113 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:46:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:46:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:40.238173+00:00 (in 29.999533 seconds)
autopruneworker stdout | 2024-04-22 03:46:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:40 UTC)" (scheduled at 2024-04-22 03:46:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:46:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757510246, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:46:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:46:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:30.247852+00:00 (in 19.998705 seconds)
namespacegcworker stdout | 2024-04-22 03:46:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:10 UTC)" (scheduled at 2024-04-22 03:46:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:46:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:46:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:46:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:46:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:46:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:46:10,637 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:46:10,787 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:46:12,191 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:46:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:22.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 03:46:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:22 UTC)" (scheduled at 2024-04-22 03:46:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:46:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:46:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 12, 431783), True, datetime.datetime(2024, 4, 22, 3, 46, 12, 431783), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:46:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:46:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:46:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:46:13,026 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:46:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:46:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:17.065745+00:00 (in 2.998139 seconds)
securityworker stdout | 2024-04-22 03:46:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:44 UTC)" (scheduled at 2024-04-22 03:46:14.067050+00:00)
securityworker stdout | 2024-04-22 03:46:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:46:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:46:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:46:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:46:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:46:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:46:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:46:14,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:46:14,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:46:14,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:46:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 41, 14, 78326), 1, 2])
securityworker stdout | 2024-04-22 03:46:14,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:46:14,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:46:14,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 41, 14, 78326), 1, 2])
securityworker stdout | 2024-04-22 03:46:14,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:46:14,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:46:14,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:14,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:46:14,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:46:14,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:46:14,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:46:14,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:14,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:46:14,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:46:15,480 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:46:15,632 [285] [DEBUG] [app] Starting request: urn:request:30a60218-af8d-4db8-97b4-cd345b6cac2e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:15,632 [282] [DEBUG] [app] Starting request: urn:request:79776069-a02d-4d18-851d-03ade8bc40b9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:46:15,636 [275] [DEBUG] [app] Starting request: urn:request:a40101ef-48fc-402e-bc42-166b2f6629ce (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:15,636 [283] [DEBUG] [app] Starting request: urn:request:fccbd98f-f591-48df-803f-656ed3a55d60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:15,636 [283] [DEBUG] [app] Ending request: urn:request:fccbd98f-f591-48df-803f-656ed3a55d60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fccbd98f-f591-48df-803f-656ed3a55d60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:15,636 [275] [DEBUG] [app] Ending request: urn:request:a40101ef-48fc-402e-bc42-166b2f6629ce (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a40101ef-48fc-402e-bc42-166b2f6629ce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:46:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:46:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:15,639 [285] [DEBUG] [app] Starting request: urn:request:fdc92f19-142d-4bc1-b1c7-5eb0904e040e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:15,640 [282] [DEBUG] [app] Starting request: urn:request:59c257a9-0814-4bb5-8336-50bf3deb909a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:15,640 [285] [DEBUG] [app] Ending request: urn:request:fdc92f19-142d-4bc1-b1c7-5eb0904e040e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdc92f19-142d-4bc1-b1c7-5eb0904e040e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:15,640 [282] [DEBUG] [app] Ending request: urn:request:59c257a9-0814-4bb5-8336-50bf3deb909a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59c257a9-0814-4bb5-8336-50bf3deb909a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:46:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:46:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:46:15,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:15,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:15,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:15,662 [282] [DEBUG] [app] Ending request: urn:request:79776069-a02d-4d18-851d-03ade8bc40b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79776069-a02d-4d18-851d-03ade8bc40b9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:15,662 [285] [DEBUG] [app] Ending request: urn:request:30a60218-af8d-4db8-97b4-cd345b6cac2e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:30a60218-af8d-4db8-97b4-cd345b6cac2e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:15,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:15,663 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:15,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:46:15,663 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
queuecleanupworker stdout | 2024-04-22 03:46:15,796 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:46:16,171 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:46:16,961 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:46:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:46:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:44.067050+00:00 (in 27.000817 seconds)
securityworker stdout | 2024-04-22 03:46:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:47 UTC)" (scheduled at 2024-04-22 03:46:17.065745+00:00)
securityworker stdout | 2024-04-22 03:46:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:46:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:46:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:46:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:46:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:46:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:46:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:46:17,953 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:46:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:46:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:53.045554+00:00 (in 34.997827 seconds)
exportactionlogsworker stdout | 2024-04-22 03:46:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:18 UTC)" (scheduled at 2024-04-22 03:46:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:46:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:46:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:46:18,863 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:46:19,372 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:46:19,501 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:46:19,504 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:46:19,507 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:46:20,528 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:46:20,924 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:46:20,943 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
builder stdout | 2024-04-22 03:46:21,006 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:46:21,022 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:46:21,025 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:46:21,030 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:46:21,034 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:46:21,038 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:46:21,044 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:46:21,049 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:46:21,054 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:46:21,209 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:46:21,212 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:46:21,215 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:46:21,219 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:46:21,222 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:46:21,796 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:46:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:32.430973+00:00 (in 9.999543 seconds)
notificationworker stdout | 2024-04-22 03:46:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:32 UTC)" (scheduled at 2024-04-22 03:46:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:46:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:46:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 22, 431704), True, datetime.datetime(2024, 4, 22, 3, 46, 22, 431704), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:46:22,452 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:46:22,452 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:46:22,452 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:46:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:46:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:24.738365+00:00 (in 59.999489 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:46:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:24 UTC)" (scheduled at 2024-04-22 03:46:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:46:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:46:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:46:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:46:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:46:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:58.130973+00:00 (in 29.999397 seconds)
buildlogsarchiver stdout | 2024-04-22 03:46:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:58 UTC)" (scheduled at 2024-04-22 03:46:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:46:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 46, 28, 131954), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:46:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:46:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:46:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:46:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:31.933721+00:00 (in 2.002140 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:29 UTC)" (scheduled at 2024-04-22 03:46:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,932 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 29, 932035), True, datetime.datetime(2024, 4, 22, 3, 46, 29, 932035), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,948 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,948 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:46:29,948 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:46:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:46:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:47.245913+00:00 (in 16.997502 seconds)
namespacegcworker stdout | 2024-04-22 03:46:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:51:30 UTC)" (scheduled at 2024-04-22 03:46:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 03:46:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 46, 30, 248804), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:46:30,263 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 46, 30, 248804), True, datetime.datetime(2024, 4, 22, 3, 46, 30, 248804), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:46:30,268 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 46, 30, 248804), True, datetime.datetime(2024, 4, 22, 3, 46, 30, 248804), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 3, 46, 30, 248804), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:46:30,272 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:46:30,272 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:51:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:46:30,632 [285] [DEBUG] [app] Starting request: urn:request:a9481217-d65d-413a-96b2-200e6be23a0b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:30,632 [286] [DEBUG] [app] Starting request: urn:request:2fc2cb9c-cde3-4ccb-9e1e-bda405c2cf12 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:46:30,636 [275] [DEBUG] [app] Starting request: urn:request:a3390d8b-43dc-47cb-ae49-0d48ae05810c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:30,636 [278] [DEBUG] [app] Starting request: urn:request:773bb010-4134-4f02-a80f-4696e14d73ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:30,637 [278] [DEBUG] [app] Ending request: urn:request:773bb010-4134-4f02-a80f-4696e14d73ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:773bb010-4134-4f02-a80f-4696e14d73ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:30,637 [275] [DEBUG] [app] Ending request: urn:request:a3390d8b-43dc-47cb-ae49-0d48ae05810c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a3390d8b-43dc-47cb-ae49-0d48ae05810c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:46:30,637 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:46:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:30,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:30,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:30,641 [284] [DEBUG] [app] Starting request: urn:request:1168f18c-1cd0-4565-ae75-9f99330dfc17 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:30,641 [282] [DEBUG] [app] Starting request: urn:request:c617a017-3c84-4a64-a747-a3335e45b3bd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:30,641 [284] [DEBUG] [app] Ending request: urn:request:1168f18c-1cd0-4565-ae75-9f99330dfc17 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1168f18c-1cd0-4565-ae75-9f99330dfc17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:30,642 [282] [DEBUG] [app] Ending request: urn:request:c617a017-3c84-4a64-a747-a3335e45b3bd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c617a017-3c84-4a64-a747-a3335e45b3bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:30,642 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:46:30,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:30,642 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:46:30,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:46:30,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:30,643 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:30,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:30,652 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:30,664 [285] [DEBUG] [app] Ending request: urn:request:a9481217-d65d-413a-96b2-200e6be23a0b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a9481217-d65d-413a-96b2-200e6be23a0b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:30,665 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:30,665 [286] [DEBUG] [app] Ending request: urn:request:2fc2cb9c-cde3-4ccb-9e1e-bda405c2cf12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2fc2cb9c-cde3-4ccb-9e1e-bda405c2cf12', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:30,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:46:30,665 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:46:30,666 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.035)
gcworker stdout | 2024-04-22 03:46:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:46:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:01.842666+00:00 (in 29.999420 seconds)
gcworker stdout | 2024-04-22 03:46:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:01 UTC)" (scheduled at 2024-04-22 03:46:31.842666+00:00)
gcworker stdout | 2024-04-22 03:46:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712547991843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:46:31,861 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:46:31,861 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:46:31,862 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:46:31,934 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:46:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:29.931002+00:00 (in 57.996559 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:46:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:31 UTC)" (scheduled at 2024-04-22 03:46:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:46:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:46:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:46:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:42.430973+00:00 (in 9.999506 seconds)
notificationworker stdout | 2024-04-22 03:46:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:42 UTC)" (scheduled at 2024-04-22 03:46:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:46:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:46:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 32, 431741), True, datetime.datetime(2024, 4, 22, 3, 46, 32, 431741), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:46:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:46:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:46:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:46:34,029 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:46:35,262 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:46:37,905 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:46:39,101 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:46:39,409 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:46:39,448 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:46:40,149 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:46:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:46:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:10.238173+00:00 (in 29.999522 seconds)
autopruneworker stdout | 2024-04-22 03:46:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:10 UTC)" (scheduled at 2024-04-22 03:46:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:46:40,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757540248, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:46:40,255 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:46:40,256 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:46:40,256 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:46:40,676 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:46:40,825 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:46:42,228 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:46:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:52.430973+00:00 (in 9.999504 seconds)
notificationworker stdout | 2024-04-22 03:46:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:52 UTC)" (scheduled at 2024-04-22 03:46:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:46:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:46:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 42, 431747), True, datetime.datetime(2024, 4, 22, 3, 46, 42, 431747), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:46:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:46:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:46:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:46:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:46:43,064 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:46:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:46:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:47.065745+00:00 (in 2.998206 seconds)
securityworker stdout | 2024-04-22 03:46:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:14 UTC)" (scheduled at 2024-04-22 03:46:44.067050+00:00)
securityworker stdout | 2024-04-22 03:46:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:46:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:46:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:46:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:46:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:46:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:46:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:46:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:46:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:46:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:46:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 41, 44, 77516), 1, 2])
securityworker stdout | 2024-04-22 03:46:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:46:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:46:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 41, 44, 77516), 1, 2])
securityworker stdout | 2024-04-22 03:46:44,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:46:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:46:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:46:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:46:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:46:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:46:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:46:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:46:44,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:46:45,518 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:46:45,632 [282] [DEBUG] [app] Starting request: urn:request:20aeeb45-d918-44b2-8946-0ed23c4eb2ff (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:45,632 [285] [DEBUG] [app] Starting request: urn:request:71d9b7c3-59c4-49db-97f5-5bf573b39134 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:46:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:46:45,636 [275] [DEBUG] [app] Starting request: urn:request:3f0ed1b4-8f45-441a-b5c1-218d569a25a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:45,636 [283] [DEBUG] [app] Starting request: urn:request:ee6e801e-5f08-4d14-bf9c-073fd326a5d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:46:45,636 [275] [DEBUG] [app] Ending request: urn:request:3f0ed1b4-8f45-441a-b5c1-218d569a25a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f0ed1b4-8f45-441a-b5c1-218d569a25a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:45,636 [283] [DEBUG] [app] Ending request: urn:request:ee6e801e-5f08-4d14-bf9c-073fd326a5d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ee6e801e-5f08-4d14-bf9c-073fd326a5d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:46:45,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:46:45,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 03:46:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:45,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:46:45,640 [285] [DEBUG] [app] Starting request: urn:request:360b7f6c-167e-4ec8-a4dc-eb9de2455d8f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:45,640 [284] [DEBUG] [app] Starting request: urn:request:26f385c4-cc3a-478b-a67e-de8cb888f9fc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:46:45,640 [285] [DEBUG] [app] Ending request: urn:request:360b7f6c-167e-4ec8-a4dc-eb9de2455d8f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:360b7f6c-167e-4ec8-a4dc-eb9de2455d8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:45,640 [284] [DEBUG] [app] Ending request: urn:request:26f385c4-cc3a-478b-a67e-de8cb888f9fc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26f385c4-cc3a-478b-a67e-de8cb888f9fc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:46:45,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:46:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:46:45,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:46:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:46:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:45,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:46:45,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:45,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:46:45,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:46:45,662 [282] [DEBUG] [app] Ending request: urn:request:20aeeb45-d918-44b2-8946-0ed23c4eb2ff (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:20aeeb45-d918-44b2-8946-0ed23c4eb2ff', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:45,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:45,663 [285] [DEBUG] [app] Ending request: urn:request:71d9b7c3-59c4-49db-97f5-5bf573b39134 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:71d9b7c3-59c4-49db-97f5-5bf573b39134', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:46:45,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:46:45,663 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:46:45,663 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:46:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:46:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
queuecleanupworker stdout | 2024-04-22 03:46:45,833 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:46:46,208 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:46:46,998 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:46:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:46:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:14.067050+00:00 (in 27.000769 seconds)
securityworker stdout | 2024-04-22 03:46:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:17 UTC)" (scheduled at 2024-04-22 03:46:47.065745+00:00)
securityworker stdout | 2024-04-22 03:46:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:46:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:46:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:46:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:46:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:46:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:46:47,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:46:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:46:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:10.248668+00:00 (in 23.002205 seconds)
namespacegcworker stdout | 2024-04-22 03:46:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:47 UTC)" (scheduled at 2024-04-22 03:46:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:46:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:46:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 47, 246723), True, datetime.datetime(2024, 4, 22, 3, 46, 47, 246723), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:46:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:46:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:46:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:46:47,990 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:46:48,902 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:46:49,411 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:46:49,511 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:46:49,515 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:46:49,519 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:46:50,565 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 03:46:50,935 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:46:50,955 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:46:51,036 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:46:51,041 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:46:51,047 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
builder stdout | 2024-04-22 03:46:51,050 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-registry stdout | 2024-04-22 03:46:51,054 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:46:51,058 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:46:51,062 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:46:51,065 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:46:51,069 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:46:51,218 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:46:51,220 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:46:51,224 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:46:51,228 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:46:51,231 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:46:51,833 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:46:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:46:54.433683+00:00 (in 2.002160 seconds)
notificationworker stdout | 2024-04-22 03:46:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:02 UTC)" (scheduled at 2024-04-22 03:46:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:46:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:46:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 52, 431982), True, datetime.datetime(2024, 4, 22, 3, 46, 52, 431982), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:46:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:46:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:46:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:46:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:46:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:18.047111+00:00 (in 25.001058 seconds)
exportactionlogsworker stdout | 2024-04-22 03:46:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:53 UTC)" (scheduled at 2024-04-22 03:46:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:46:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:46:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 46, 53, 46501), True, datetime.datetime(2024, 4, 22, 3, 46, 53, 46501), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:46:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:46:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:46:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:46:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:46:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:02.430973+00:00 (in 7.996779 seconds)
notificationworker stdout | 2024-04-22 03:46:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:54 UTC)" (scheduled at 2024-04-22 03:46:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:46:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:46:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:47:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:46:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:46:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:28.130973+00:00 (in 29.999477 seconds)
buildlogsarchiver stdout | 2024-04-22 03:46:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:28 UTC)" (scheduled at 2024-04-22 03:46:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:46:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 46, 58, 131869), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:46:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:46:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:46:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:47:00,632 [282] [DEBUG] [app] Starting request: urn:request:fd592468-01de-4633-b2d4-1c475a4744ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:00,632 [284] [DEBUG] [app] Starting request: urn:request:5ae7388e-bfac-4470-9ad0-ca778786087b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:47:00,636 [281] [DEBUG] [app] Starting request: urn:request:ea55c74c-d04d-4809-abb9-f4c873e72080 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:00,636 [283] [DEBUG] [app] Starting request: urn:request:193753c9-e35b-496c-a9b3-111abf5e668c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:00,636 [281] [DEBUG] [app] Ending request: urn:request:ea55c74c-d04d-4809-abb9-f4c873e72080 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea55c74c-d04d-4809-abb9-f4c873e72080', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:00,636 [283] [DEBUG] [app] Ending request: urn:request:193753c9-e35b-496c-a9b3-111abf5e668c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:193753c9-e35b-496c-a9b3-111abf5e668c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:47:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:00,640 [282] [DEBUG] [app] Starting request: urn:request:d4de5951-cc2f-4624-8b1e-a975830d273c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:00,640 [285] [DEBUG] [app] Starting request: urn:request:d49eb9b8-7694-4ed3-b6c0-7df744c01526 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:00,640 [282] [DEBUG] [app] Ending request: urn:request:d4de5951-cc2f-4624-8b1e-a975830d273c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d4de5951-cc2f-4624-8b1e-a975830d273c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:00,641 [285] [DEBUG] [app] Ending request: urn:request:d49eb9b8-7694-4ed3-b6c0-7df744c01526 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d49eb9b8-7694-4ed3-b6c0-7df744c01526', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:00,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:47:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:00,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:00,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:00,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:00,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:00,663 [282] [DEBUG] [app] Ending request: urn:request:fd592468-01de-4633-b2d4-1c475a4744ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd592468-01de-4633-b2d4-1c475a4744ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:47:00,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:00,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:47:00,663 [284] [DEBUG] [app] Ending request: urn:request:5ae7388e-bfac-4470-9ad0-ca778786087b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5ae7388e-bfac-4470-9ad0-ca778786087b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:47:00,664 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:00,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gcworker stdout | 2024-04-22 03:47:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:47:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:31.842666+00:00 (in 29.999473 seconds)
gcworker stdout | 2024-04-22 03:47:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:31 UTC)" (scheduled at 2024-04-22 03:47:01.842666+00:00)
gcworker stdout | 2024-04-22 03:47:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548021843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:47:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:47:01,861 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:47:01,861 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:47:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:47:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:09.033206+00:00 (in 6.996930 seconds)
repositorygcworker stdout | 2024-04-22 03:47:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:02 UTC)" (scheduled at 2024-04-22 03:47:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:47:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:47:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:47:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:12.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 03:47:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:12 UTC)" (scheduled at 2024-04-22 03:47:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:47:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:47:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 2, 431873), True, datetime.datetime(2024, 4, 22, 3, 47, 2, 431873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:47:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:47:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:47:04,066 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:47:05,298 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:47:07,916 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:47:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:47:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:02.035613+00:00 (in 53.001873 seconds)
repositorygcworker stdout | 2024-04-22 03:47:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:09 UTC)" (scheduled at 2024-04-22 03:47:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:47:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:47:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 9, 34002), True, datetime.datetime(2024, 4, 22, 3, 47, 9, 34002), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:47:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:47:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:47:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:47:09,111 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:47:09,446 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:47:09,484 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:47:10,187 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:47:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:47:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:40.238173+00:00 (in 29.999526 seconds)
autopruneworker stdout | 2024-04-22 03:47:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:40 UTC)" (scheduled at 2024-04-22 03:47:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:47:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757570247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:47:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:47:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:47.245913+00:00 (in 36.996729 seconds)
namespacegcworker stdout | 2024-04-22 03:47:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:10 UTC)" (scheduled at 2024-04-22 03:47:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:47:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:47:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:47:10,255 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:47:10,255 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:47:10,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:40 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:47:10,714 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:47:10,863 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:47:12,265 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:47:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:22.430973+00:00 (in 9.999545 seconds)
notificationworker stdout | 2024-04-22 03:47:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:22 UTC)" (scheduled at 2024-04-22 03:47:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:47:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:47:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 12, 431706), True, datetime.datetime(2024, 4, 22, 3, 47, 12, 431706), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:47:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:47:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:47:13,101 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:47:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:47:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:17.065745+00:00 (in 2.998123 seconds)
securityworker stdout | 2024-04-22 03:47:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:44 UTC)" (scheduled at 2024-04-22 03:47:14.067050+00:00)
securityworker stdout | 2024-04-22 03:47:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:47:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:47:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:47:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:47:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:47:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:47:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:47:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:47:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:47:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:47:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 42, 14, 77794), 1, 2])
securityworker stdout | 2024-04-22 03:47:14,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:47:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:47:14,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 42, 14, 77794), 1, 2])
securityworker stdout | 2024-04-22 03:47:14,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:47:14,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:47:14,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:14,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:47:14,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:47:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:47:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:47:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:47:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:47:15,555 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:47:15,632 [282] [DEBUG] [app] Starting request: urn:request:c26c2864-ce1f-4d54-923b-ba4159a0ac05 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:15,632 [286] [DEBUG] [app] Starting request: urn:request:9708d06e-0aa1-4a38-bebf-5d36b8764a32 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:47:15,636 [275] [DEBUG] [app] Starting request: urn:request:b618e4f5-6d74-4ee4-a433-8c9c0db22f27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:15,636 [281] [DEBUG] [app] Starting request: urn:request:97f17fe6-c382-4413-8461-7c1c6a25b203 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:15,636 [275] [DEBUG] [app] Ending request: urn:request:b618e4f5-6d74-4ee4-a433-8c9c0db22f27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b618e4f5-6d74-4ee4-a433-8c9c0db22f27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:15,637 [281] [DEBUG] [app] Ending request: urn:request:97f17fe6-c382-4413-8461-7c1c6a25b203 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97f17fe6-c382-4413-8461-7c1c6a25b203', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:47:15,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:47:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:15,640 [284] [DEBUG] [app] Starting request: urn:request:8e78f39b-602b-4d3e-9129-c178683ab954 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:15,640 [285] [DEBUG] [app] Starting request: urn:request:025091ce-51c0-45e1-b48b-ca43e03d9aa3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:15,640 [284] [DEBUG] [app] Ending request: urn:request:8e78f39b-602b-4d3e-9129-c178683ab954 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e78f39b-602b-4d3e-9129-c178683ab954', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:15,641 [285] [DEBUG] [app] Ending request: urn:request:025091ce-51c0-45e1-b48b-ca43e03d9aa3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:025091ce-51c0-45e1-b48b-ca43e03d9aa3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:47:15,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:15,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:47:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:15,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:15,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:15,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:15,663 [282] [DEBUG] [app] Ending request: urn:request:c26c2864-ce1f-4d54-923b-ba4159a0ac05 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c26c2864-ce1f-4d54-923b-ba4159a0ac05', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:47:15,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:15,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:47:15,663 [286] [DEBUG] [app] Ending request: urn:request:9708d06e-0aa1-4a38-bebf-5d36b8764a32 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9708d06e-0aa1-4a38-bebf-5d36b8764a32', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:47:15,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:15,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
queuecleanupworker stdout | 2024-04-22 03:47:15,870 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:47:16,245 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 03:47:17,035 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:47:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:47:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:44.067050+00:00 (in 27.000761 seconds)
securityworker stdout | 2024-04-22 03:47:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:47 UTC)" (scheduled at 2024-04-22 03:47:17.065745+00:00)
securityworker stdout | 2024-04-22 03:47:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:47:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:47:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:47:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:47:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:47:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:47:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:47:18,027 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
exportactionlogsworker stdout | 2024-04-22 03:47:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:47:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:23.046900+00:00 (in 4.999206 seconds)
exportactionlogsworker stdout | 2024-04-22 03:47:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:18 UTC)" (scheduled at 2024-04-22 03:47:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:47:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:47:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:18 UTC)" executed successfully
securityworker stdout | 2024-04-22 03:47:18,917 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:47:19,237 [73] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
manifestbackfillworker stdout | 2024-04-22 03:47:19,238 [73] [INFO] [apscheduler.executors.default] Running job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2024-04-22 03:47:19 UTC)" (scheduled at 2024-04-22 03:47:19.237742+00:00)
manifestbackfillworker stdout | 2024-04-22 03:47:19,239 [73] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."layers_compressed_size" IS %s) LIMIT %s OFFSET %s', [None, 1, 0])
manifestbackfillworker stdout | 2024-04-22 03:47:19,239 [73] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:47:19.237742+00:00 (in 3599.997924 seconds)
manifestbackfillworker stdout | 2024-04-22 03:47:19,253 [73] [DEBUG] [__main__] Manifest backfill worker has completed; skipping
manifestbackfillworker stdout | 2024-04-22 03:47:19,253 [73] [DEBUG] [data.database] Disconnecting from database.
manifestbackfillworker stdout | 2024-04-22 03:47:19,253 [73] [INFO] [apscheduler.executors.default] Job "ManifestBackfillWorker._backfill_manifests (trigger: interval[1:00:00], next run at: 2024-04-22 04:47:19 UTC)" executed successfully
manifestbackfillworker stdout | 2024-04-22 03:47:19,449 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:47:19,522 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:47:19,525 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:47:19,529 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:47:20,603 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:47:20,966 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
namespacegcworker stdout | 2024-04-22 03:47:20,971 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:47:21,049 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:47:21,052 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:47:21,059 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:47:21,066 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:47:21,070 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:47:21,073 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:47:21,076 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:47:21,082 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
builder stdout | 2024-04-22 03:47:21,086 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-web stdout | 2024-04-22 03:47:21,229 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:47:21,234 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:47:21,239 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:47:21,242 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:47:21,245 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:47:21,870 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:47:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:32.430973+00:00 (in 9.999515 seconds)
notificationworker stdout | 2024-04-22 03:47:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:32 UTC)" (scheduled at 2024-04-22 03:47:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:47:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:47:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 22, 431736), True, datetime.datetime(2024, 4, 22, 3, 47, 22, 431736), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:47:22,455 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:47:22,455 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:22,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:32 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:47:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:47:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:53.045554+00:00 (in 29.998024 seconds)
exportactionlogsworker stdout | 2024-04-22 03:47:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:52:23 UTC)" (scheduled at 2024-04-22 03:47:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 03:47:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 47, 23, 47777), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:47:23,063 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 47, 23, 47777), True, datetime.datetime(2024, 4, 22, 3, 47, 23, 47777), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:47:23,068 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 47, 23, 47777), True, datetime.datetime(2024, 4, 22, 3, 47, 23, 47777), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 3, 47, 23, 47777), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:47:23,072 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:47:23,072 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:52:23 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:47:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:47:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:24.738365+00:00 (in 59.999465 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:47:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:24 UTC)" (scheduled at 2024-04-22 03:47:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:47:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:47:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:47:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:47:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:47:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:58.130973+00:00 (in 29.999439 seconds)
buildlogsarchiver stdout | 2024-04-22 03:47:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:58 UTC)" (scheduled at 2024-04-22 03:47:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:47:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 47, 28, 131905), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:47:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:47:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:47:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:47:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:31.933721+00:00 (in 2.002151 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:29 UTC)" (scheduled at 2024-04-22 03:47:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 29, 931865), True, datetime.datetime(2024, 4, 22, 3, 47, 29, 931865), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:47:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:47:30,632 [282] [DEBUG] [app] Starting request: urn:request:3e334a3f-be0c-49c9-8ae1-9ad7fef6ea2b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:30,632 [286] [DEBUG] [app] Starting request: urn:request:d00031e5-0d1c-477b-9a27-8ac7157a2d49 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:47:30,636 [280] [DEBUG] [app] Starting request: urn:request:502b6fb0-7559-4d8b-856e-108cfd5b30c7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:30,636 [275] [DEBUG] [app] Starting request: urn:request:88511901-a31f-408e-8f06-f42a87c708d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:30,637 [275] [DEBUG] [app] Ending request: urn:request:88511901-a31f-408e-8f06-f42a87c708d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:88511901-a31f-408e-8f06-f42a87c708d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:30,637 [280] [DEBUG] [app] Ending request: urn:request:502b6fb0-7559-4d8b-856e-108cfd5b30c7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:502b6fb0-7559-4d8b-856e-108cfd5b30c7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:47:30,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:47:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:30,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:30,640 [284] [DEBUG] [app] Starting request: urn:request:32074b55-04c0-42fd-9b64-6254889ef0b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:30,640 [282] [DEBUG] [app] Starting request: urn:request:d9253ff6-f01e-46cd-a02a-f15d46940e30 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:30,641 [284] [DEBUG] [app] Ending request: urn:request:32074b55-04c0-42fd-9b64-6254889ef0b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32074b55-04c0-42fd-9b64-6254889ef0b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:30,641 [282] [DEBUG] [app] Ending request: urn:request:d9253ff6-f01e-46cd-a02a-f15d46940e30 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9253ff6-f01e-46cd-a02a-f15d46940e30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:30,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:30,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:30,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:47:30,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:30,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:30,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:30,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:30,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:30,663 [282] [DEBUG] [app] Ending request: urn:request:3e334a3f-be0c-49c9-8ae1-9ad7fef6ea2b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3e334a3f-be0c-49c9-8ae1-9ad7fef6ea2b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:47:30,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:30,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:47:30,664 [286] [DEBUG] [app] Ending request: urn:request:d00031e5-0d1c-477b-9a27-8ac7157a2d49 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d00031e5-0d1c-477b-9a27-8ac7157a2d49', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:47:30,665 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:47:30,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:47:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:47:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:01.842666+00:00 (in 29.999445 seconds)
gcworker stdout | 2024-04-22 03:47:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:01 UTC)" (scheduled at 2024-04-22 03:47:31.842666+00:00)
gcworker stdout | 2024-04-22 03:47:31,845 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548051843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:47:31,861 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:47:31,861 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:47:31,862 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:47:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:47:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:29.931002+00:00 (in 57.996686 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:47:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:31 UTC)" (scheduled at 2024-04-22 03:47:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:47:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:47:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:47:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:38.433394+00:00 (in 6.001836 seconds)
notificationworker stdout | 2024-04-22 03:47:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:42 UTC)" (scheduled at 2024-04-22 03:47:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:47:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:47:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 32, 431868), True, datetime.datetime(2024, 4, 22, 3, 47, 32, 431868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:47:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:47:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:47:34,103 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:47:35,337 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:47:37,953 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
notificationworker stdout | 2024-04-22 03:47:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:42.430973+00:00 (in 3.997098 seconds)
notificationworker stdout | 2024-04-22 03:47:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:52:38 UTC)" (scheduled at 2024-04-22 03:47:38.433394+00:00)
notificationworker stdout | 2024-04-22 03:47:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 47, 38, 434250), 'notification/%'])
notificationworker stdout | 2024-04-22 03:47:38,449 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 47, 38, 434250), True, datetime.datetime(2024, 4, 22, 3, 47, 38, 434250), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 03:47:38,453 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 47, 38, 434250), True, datetime.datetime(2024, 4, 22, 3, 47, 38, 434250), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 3, 47, 38, 434250), 'notification/%'])
notificationworker stdout | 2024-04-22 03:47:38,457 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:38,457 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:52:38 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:47:39,119 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:47:39,483 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:47:39,519 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 03:47:40,224 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
autopruneworker stdout | 2024-04-22 03:47:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:47:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:10.238173+00:00 (in 29.999483 seconds)
autopruneworker stdout | 2024-04-22 03:47:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:10 UTC)" (scheduled at 2024-04-22 03:47:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:47:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757600247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:47:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:47:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:47:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:10 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 03:47:40,751 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:47:40,900 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:47:42,304 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:47:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:52.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 03:47:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:52 UTC)" (scheduled at 2024-04-22 03:47:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:47:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:47:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 42, 431812), True, datetime.datetime(2024, 4, 22, 3, 47, 42, 431812), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:47:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:47:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:47:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:47:43,139 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:47:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:47:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:47.065745+00:00 (in 2.998195 seconds)
securityworker stdout | 2024-04-22 03:47:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:14 UTC)" (scheduled at 2024-04-22 03:47:44.067050+00:00)
securityworker stdout | 2024-04-22 03:47:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:47:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:47:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:47:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:47:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:47:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:47:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:47:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:47:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:47:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:47:44,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 42, 44, 78326), 1, 2])
securityworker stdout | 2024-04-22 03:47:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:47:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:47:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 42, 44, 78326), 1, 2])
securityworker stdout | 2024-04-22 03:47:44,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:47:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:47:44,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:44,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:47:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:47:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:47:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:47:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:47:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:47:44,109 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:14 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:47:45,592 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:47:45,633 [282] [DEBUG] [app] Starting request: urn:request:de141288-6856-4669-93ea-dbb8690a31dc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:45,633 [286] [DEBUG] [app] Starting request: urn:request:e2fec31c-32bb-4607-9919-32de4df42d25 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:47:45,635 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:45,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:47:45,637 [283] [DEBUG] [app] Starting request: urn:request:353e81fb-035a-4a93-8b4f-7180dd9aa8b4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:45,637 [281] [DEBUG] [app] Starting request: urn:request:b92ddd0e-e1ca-4aa2-9fcc-9d7f46a550f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:47:45,637 [281] [DEBUG] [app] Ending request: urn:request:b92ddd0e-e1ca-4aa2-9fcc-9d7f46a550f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b92ddd0e-e1ca-4aa2-9fcc-9d7f46a550f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:45,637 [283] [DEBUG] [app] Ending request: urn:request:353e81fb-035a-4a93-8b4f-7180dd9aa8b4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:353e81fb-035a-4a93-8b4f-7180dd9aa8b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:47:45,638 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:47:45,638 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:47:45,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:47:45,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:45,640 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:45,640 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:47:45,641 [286] [DEBUG] [app] Starting request: urn:request:45d995e7-2a55-4aaa-8241-98edbcf0e015 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:45,642 [284] [DEBUG] [app] Starting request: urn:request:37c18a4c-f346-4dec-b0b0-3fc2fb9d2cfb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:47:45,642 [286] [DEBUG] [app] Ending request: urn:request:45d995e7-2a55-4aaa-8241-98edbcf0e015 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:45d995e7-2a55-4aaa-8241-98edbcf0e015', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:45,642 [284] [DEBUG] [app] Ending request: urn:request:37c18a4c-f346-4dec-b0b0-3fc2fb9d2cfb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:37c18a4c-f346-4dec-b0b0-3fc2fb9d2cfb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:47:45,642 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:47:45,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:45,642 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:47:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:47:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:47:45,643 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:47:45,643 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:45,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:47:45,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:45,652 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:47:45,664 [286] [DEBUG] [app] Ending request: urn:request:e2fec31c-32bb-4607-9919-32de4df42d25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2fec31c-32bb-4607-9919-32de4df42d25', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:47:45,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:45,664 [282] [DEBUG] [app] Ending request: urn:request:de141288-6856-4669-93ea-dbb8690a31dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de141288-6856-4669-93ea-dbb8690a31dc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:47:45,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:47:45,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:47:45,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:47:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:47:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.033)
queuecleanupworker stdout | 2024-04-22 03:47:45,888 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:47:46,283 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:47:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:47:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:14.067050+00:00 (in 27.000816 seconds)
securityworker stdout | 2024-04-22 03:47:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:17 UTC)" (scheduled at 2024-04-22 03:47:47.065745+00:00)
securityworker stdout | 2024-04-22 03:47:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:47:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:47:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:47:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
exportactionlogsworker stdout | 2024-04-22 03:47:47,072 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
securityworker stdout | 2024-04-22 03:47:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:47:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:47:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:47:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:47:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:10.248668+00:00 (in 23.002158 seconds)
namespacegcworker stdout | 2024-04-22 03:47:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:47 UTC)" (scheduled at 2024-04-22 03:47:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:47:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:47:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 47, 246784), True, datetime.datetime(2024, 4, 22, 3, 47, 47, 246784), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:47:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:47:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:47:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:47:48,064 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:47:48,954 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:47:49,486 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:47:49,534 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:47:49,538 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:47:49,541 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:47:50,640 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:47:50,979 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
namespacegcworker stdout | 2024-04-22 03:47:51,008 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:47:51,061 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:47:51,064 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:47:51,067 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:47:51,078 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:47:51,082 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:47:51,085 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:47:51,088 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:47:51,092 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
builder stdout | 2024-04-22 03:47:51,122 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-web stdout | 2024-04-22 03:47:51,238 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:47:51,241 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:47:51,247 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:47:51,250 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:47:51,253 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:47:51,907 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:47:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:47:54.433683+00:00 (in 2.002201 seconds)
notificationworker stdout | 2024-04-22 03:47:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:02 UTC)" (scheduled at 2024-04-22 03:47:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:47:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:47:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 52, 431924), True, datetime.datetime(2024, 4, 22, 3, 47, 52, 431924), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:47:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:47:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:47:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:47:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:47:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:18.047111+00:00 (in 25.001047 seconds)
exportactionlogsworker stdout | 2024-04-22 03:47:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:53 UTC)" (scheduled at 2024-04-22 03:47:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:47:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:47:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 47, 53, 46408), True, datetime.datetime(2024, 4, 22, 3, 47, 53, 46408), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:47:53,069 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:47:53,070 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:47:53,070 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:47:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:47:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:02.430973+00:00 (in 7.996719 seconds)
notificationworker stdout | 2024-04-22 03:47:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:54 UTC)" (scheduled at 2024-04-22 03:47:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:47:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:47:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:48:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:47:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:47:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:28.130973+00:00 (in 29.999501 seconds)
buildlogsarchiver stdout | 2024-04-22 03:47:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:28 UTC)" (scheduled at 2024-04-22 03:47:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:47:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 47, 58, 131883), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:47:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:47:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:47:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:48:00,632 [285] [DEBUG] [app] Starting request: urn:request:a3592ed5-33cc-45d6-a194-a3d6ab93d458 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:00,632 [286] [DEBUG] [app] Starting request: urn:request:5374644e-33d8-4d95-83ec-744b4a84734f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:48:00,636 [283] [DEBUG] [app] Starting request: urn:request:454daf7d-efc5-444a-acce-b245e0b45176 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:00,636 [278] [DEBUG] [app] Starting request: urn:request:df3aba9d-5973-44fb-a52f-0210bfc05f8d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:00,637 [283] [DEBUG] [app] Ending request: urn:request:454daf7d-efc5-444a-acce-b245e0b45176 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:454daf7d-efc5-444a-acce-b245e0b45176', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:00,637 [278] [DEBUG] [app] Ending request: urn:request:df3aba9d-5973-44fb-a52f-0210bfc05f8d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:df3aba9d-5973-44fb-a52f-0210bfc05f8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:48:00,637 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:48:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:00,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:00,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:00,640 [286] [DEBUG] [app] Starting request: urn:request:60075e2c-3480-4f4f-90d5-aca306b1cb2e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:00,641 [285] [DEBUG] [app] Starting request: urn:request:4f997e1b-c68f-42ed-9177-15088c12e72f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:00,641 [286] [DEBUG] [app] Ending request: urn:request:60075e2c-3480-4f4f-90d5-aca306b1cb2e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:60075e2c-3480-4f4f-90d5-aca306b1cb2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:00,641 [285] [DEBUG] [app] Ending request: urn:request:4f997e1b-c68f-42ed-9177-15088c12e72f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f997e1b-c68f-42ed-9177-15088c12e72f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:48:00,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:00,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:00,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:00,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:00,664 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:00,664 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:00,677 [285] [DEBUG] [app] Ending request: urn:request:a3592ed5-33cc-45d6-a194-a3d6ab93d458 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a3592ed5-33cc-45d6-a194-a3d6ab93d458', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:00,677 [286] [DEBUG] [app] Ending request: urn:request:5374644e-33d8-4d95-83ec-744b4a84734f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5374644e-33d8-4d95-83ec-744b4a84734f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:00,677 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:00,677 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:00,677 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:48:00,678 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.046 118 0.046)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.046 118 0.047)
gcworker stdout | 2024-04-22 03:48:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:48:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:31.842666+00:00 (in 29.999416 seconds)
gcworker stdout | 2024-04-22 03:48:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:31 UTC)" (scheduled at 2024-04-22 03:48:01.842666+00:00)
gcworker stdout | 2024-04-22 03:48:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548081843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:48:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:48:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:48:01,861 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:48:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:48:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:09.033206+00:00 (in 6.997051 seconds)
repositorygcworker stdout | 2024-04-22 03:48:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:02 UTC)" (scheduled at 2024-04-22 03:48:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:48:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:48:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:48:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:12.430973+00:00 (in 9.999459 seconds)
notificationworker stdout | 2024-04-22 03:48:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:12 UTC)" (scheduled at 2024-04-22 03:48:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:48:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:48:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 2, 431852), True, datetime.datetime(2024, 4, 22, 3, 48, 2, 431852), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:48:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:48:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:48:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:48:04,141 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:48:05,376 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:48:07,991 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:48:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:48:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:26.035364+00:00 (in 17.001673 seconds)
repositorygcworker stdout | 2024-04-22 03:48:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:09 UTC)" (scheduled at 2024-04-22 03:48:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:48:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:48:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 9, 33957), True, datetime.datetime(2024, 4, 22, 3, 48, 9, 33957), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:48:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:48:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:48:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:48:09,156 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:48:09,520 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:48:09,555 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:48:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:48:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:40.238173+00:00 (in 29.999447 seconds)
autopruneworker stdout | 2024-04-22 03:48:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:40 UTC)" (scheduled at 2024-04-22 03:48:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:48:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757630247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:48:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:48:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:47.245913+00:00 (in 36.996761 seconds)
namespacegcworker stdout | 2024-04-22 03:48:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:10 UTC)" (scheduled at 2024-04-22 03:48:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:48:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:48:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:48:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:48:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:48:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:48:10,260 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:48:10,788 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:48:10,937 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:48:12,341 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:48:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:22.430973+00:00 (in 9.999581 seconds)
notificationworker stdout | 2024-04-22 03:48:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:22 UTC)" (scheduled at 2024-04-22 03:48:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:48:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:48:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 12, 431690), True, datetime.datetime(2024, 4, 22, 3, 48, 12, 431690), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:48:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:48:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:48:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:48:13,176 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:48:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:48:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:17.065745+00:00 (in 2.998107 seconds)
securityworker stdout | 2024-04-22 03:48:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:44 UTC)" (scheduled at 2024-04-22 03:48:14.067050+00:00)
securityworker stdout | 2024-04-22 03:48:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:48:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:48:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:48:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:48:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:48:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:48:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:48:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:48:14,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:48:14,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:48:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 43, 14, 77718), 1, 2])
securityworker stdout | 2024-04-22 03:48:14,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:48:14,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:48:14,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 43, 14, 77718), 1, 2])
securityworker stdout | 2024-04-22 03:48:14,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:48:14,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:48:14,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:14,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:48:14,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:48:14,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:48:14,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:48:14,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:14,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:48:14,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:44 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 03:48:15,629 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
gunicorn-web stdout | 2024-04-22 03:48:15,632 [285] [DEBUG] [app] Starting request: urn:request:f79cec67-ce39-4b77-8724-16460719f469 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:15,632 [282] [DEBUG] [app] Starting request: urn:request:df759097-7305-48d4-93d5-56b9ef5885f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:48:15,635 [275] [DEBUG] [app] Starting request: urn:request:53040d13-636a-4024-823d-0aaab8ebda11 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:15,636 [281] [DEBUG] [app] Starting request: urn:request:a82978b8-e52c-4944-aec6-a579decd400b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:15,636 [275] [DEBUG] [app] Ending request: urn:request:53040d13-636a-4024-823d-0aaab8ebda11 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:53040d13-636a-4024-823d-0aaab8ebda11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:15,636 [281] [DEBUG] [app] Ending request: urn:request:a82978b8-e52c-4944-aec6-a579decd400b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a82978b8-e52c-4944-aec6-a579decd400b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:48:15,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 03:48:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:15,640 [285] [DEBUG] [app] Starting request: urn:request:280dca98-0598-4b97-b94c-3fe5743bf0c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:15,640 [286] [DEBUG] [app] Starting request: urn:request:7f377a06-cdeb-4201-a9e9-8ed684916eec (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:15,640 [285] [DEBUG] [app] Ending request: urn:request:280dca98-0598-4b97-b94c-3fe5743bf0c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:280dca98-0598-4b97-b94c-3fe5743bf0c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:15,640 [286] [DEBUG] [app] Ending request: urn:request:7f377a06-cdeb-4201-a9e9-8ed684916eec (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f377a06-cdeb-4201-a9e9-8ed684916eec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:48:15,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:15,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:15,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:15,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:15,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:15,662 [282] [DEBUG] [app] Ending request: urn:request:df759097-7305-48d4-93d5-56b9ef5885f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:df759097-7305-48d4-93d5-56b9ef5885f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:15,662 [285] [DEBUG] [app] Ending request: urn:request:f79cec67-ce39-4b77-8724-16460719f469 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f79cec67-ce39-4b77-8724-16460719f469', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:15,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:48:15,663 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:15,663 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
queuecleanupworker stdout | 2024-04-22 03:48:15,928 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:48:16,320 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:48:17,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:48:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:44.067050+00:00 (in 27.000745 seconds)
securityworker stdout | 2024-04-22 03:48:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:47 UTC)" (scheduled at 2024-04-22 03:48:17.065745+00:00)
securityworker stdout | 2024-04-22 03:48:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:48:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:48:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:48:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:48:17,084 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:48:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:48:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:48:17,109 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:48:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:48:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:53.045554+00:00 (in 34.997822 seconds)
exportactionlogsworker stdout | 2024-04-22 03:48:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:18 UTC)" (scheduled at 2024-04-22 03:48:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:48:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:48:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:48:18,097 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:48:18,991 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:48:19,524 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:48:19,545 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:48:19,548 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:48:19,551 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:48:20,677 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:48:20,989 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
namespacegcworker stdout | 2024-04-22 03:48:21,045 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:48:21,073 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:48:21,077 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:48:21,080 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:48:21,087 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:48:21,092 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:48:21,096 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:48:21,099 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:48:21,102 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
builder stdout | 2024-04-22 03:48:21,158 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-web stdout | 2024-04-22 03:48:21,246 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:48:21,249 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:48:21,256 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:48:21,259 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:48:21,262 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:48:21,944 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:48:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:32.430973+00:00 (in 9.999508 seconds)
notificationworker stdout | 2024-04-22 03:48:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:32 UTC)" (scheduled at 2024-04-22 03:48:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:48:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:48:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 22, 431728), True, datetime.datetime(2024, 4, 22, 3, 48, 22, 431728), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:48:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:48:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:48:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:48:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:48:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:24.738365+00:00 (in 59.999457 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:48:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:24 UTC)" (scheduled at 2024-04-22 03:48:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:48:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:48:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:48:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:48:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:48:26,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:02.035613+00:00 (in 35.999629 seconds)
repositorygcworker stdout | 2024-04-22 03:48:26,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:53:26 UTC)" (scheduled at 2024-04-22 03:48:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 03:48:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 48, 26, 36258), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:48:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 48, 26, 36258), True, datetime.datetime(2024, 4, 22, 3, 48, 26, 36258), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:48:26,055 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 48, 26, 36258), True, datetime.datetime(2024, 4, 22, 3, 48, 26, 36258), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 3, 48, 26, 36258), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:48:26,059 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:48:26,059 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:53:26 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:48:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:48:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:58.130973+00:00 (in 29.999453 seconds)
buildlogsarchiver stdout | 2024-04-22 03:48:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:58 UTC)" (scheduled at 2024-04-22 03:48:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:48:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 48, 28, 131811), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:48:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:48:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:48:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:48:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:30.933445+00:00 (in 1.001960 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:29 UTC)" (scheduled at 2024-04-22 03:48:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 29, 931790), True, datetime.datetime(2024, 4, 22, 3, 48, 29, 931790), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:48:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:48:30,632 [284] [DEBUG] [app] Starting request: urn:request:53b7a803-f8c7-418b-a28f-556d752719d7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:30,632 [285] [DEBUG] [app] Starting request: urn:request:59ae6bc1-eabb-40d2-985b-5f0a6cc8b100 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:48:30,636 [275] [DEBUG] [app] Starting request: urn:request:750121db-e7b3-4f57-9548-26c51144725c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:30,636 [283] [DEBUG] [app] Starting request: urn:request:76f8aa36-3abc-4b96-a667-a1ee405b9463 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:30,637 [275] [DEBUG] [app] Ending request: urn:request:750121db-e7b3-4f57-9548-26c51144725c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:750121db-e7b3-4f57-9548-26c51144725c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:30,637 [283] [DEBUG] [app] Ending request: urn:request:76f8aa36-3abc-4b96-a667-a1ee405b9463 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76f8aa36-3abc-4b96-a667-a1ee405b9463', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:48:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:48:30,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:30,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:30,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:30,639 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:30,640 [285] [DEBUG] [app] Starting request: urn:request:a7968844-a788-4183-8f66-e6573cb8720e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:30,641 [285] [DEBUG] [app] Ending request: urn:request:a7968844-a788-4183-8f66-e6573cb8720e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7968844-a788-4183-8f66-e6573cb8720e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:48:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:30,641 [282] [DEBUG] [app] Starting request: urn:request:fa931f02-6d06-456c-825e-679bdee8579c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:30,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:30,642 [282] [DEBUG] [app] Ending request: urn:request:fa931f02-6d06-456c-825e-679bdee8579c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fa931f02-6d06-456c-825e-679bdee8579c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:30,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:30,642 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:30,642 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:48:30,643 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:30,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:30,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:30,663 [284] [DEBUG] [app] Ending request: urn:request:53b7a803-f8c7-418b-a28f-556d752719d7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:53b7a803-f8c7-418b-a28f-556d752719d7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:30,663 [285] [DEBUG] [app] Ending request: urn:request:59ae6bc1-eabb-40d2-985b-5f0a6cc8b100 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:59ae6bc1-eabb-40d2-985b-5f0a6cc8b100', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:30,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:30,663 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:30,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:48:30,664 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.032)
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:31.933721+00:00 (in 0.999788 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:53:30 UTC)" (scheduled at 2024-04-22 03:48:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 48, 30, 934244), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,948 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 48, 30, 934244), True, datetime.datetime(2024, 4, 22, 3, 48, 30, 934244), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,953 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 48, 30, 934244), True, datetime.datetime(2024, 4, 22, 3, 48, 30, 934244), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 3, 48, 30, 934244), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,957 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:48:30,958 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:53:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:48:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:48:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:01.842666+00:00 (in 29.999471 seconds)
gcworker stdout | 2024-04-22 03:48:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:01 UTC)" (scheduled at 2024-04-22 03:48:31.842666+00:00)
gcworker stdout | 2024-04-22 03:48:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548111843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:48:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:48:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:48:31,861 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:48:31,934 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:48:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:29.931002+00:00 (in 57.996727 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:48:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:31 UTC)" (scheduled at 2024-04-22 03:48:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:48:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:48:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:48:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:42.430973+00:00 (in 9.999538 seconds)
notificationworker stdout | 2024-04-22 03:48:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:42 UTC)" (scheduled at 2024-04-22 03:48:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:48:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:48:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 32, 431700), True, datetime.datetime(2024, 4, 22, 3, 48, 32, 431700), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:48:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:48:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:48:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:48:34,162 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:48:35,413 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:48:38,028 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:48:39,193 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:48:39,557 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:48:39,590 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:48:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:48:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:10.238173+00:00 (in 29.999294 seconds)
autopruneworker stdout | 2024-04-22 03:48:40,239 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:10 UTC)" (scheduled at 2024-04-22 03:48:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:48:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757660247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:48:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:48:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:48:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:48:40,268 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:48:40,825 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:48:40,973 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:48:42,377 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:48:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:52.430973+00:00 (in 9.999535 seconds)
notificationworker stdout | 2024-04-22 03:48:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:52 UTC)" (scheduled at 2024-04-22 03:48:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:48:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:48:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 42, 431793), True, datetime.datetime(2024, 4, 22, 3, 48, 42, 431793), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:48:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:48:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:48:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:48:52 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:48:43,212 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:48:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:48:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:47.065745+00:00 (in 2.998193 seconds)
securityworker stdout | 2024-04-22 03:48:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:14 UTC)" (scheduled at 2024-04-22 03:48:44.067050+00:00)
securityworker stdout | 2024-04-22 03:48:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:48:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:48:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:48:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:48:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:48:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:48:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:48:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:48:44,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:48:44,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:48:44,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 43, 44, 77957), 1, 2])
securityworker stdout | 2024-04-22 03:48:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:48:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:48:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 43, 44, 77957), 1, 2])
securityworker stdout | 2024-04-22 03:48:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:48:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:48:44,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:44,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:48:44,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:48:44,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:48:44,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:48:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:48:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:48:44,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:48:45,632 [286] [DEBUG] [app] Starting request: urn:request:ed9d6377-3cc2-4bbc-a198-e96d6d694d3c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:45,632 [282] [DEBUG] [app] Starting request: urn:request:27b1a75d-df60-44ca-a8a8-06bc3d7debbd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:48:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:48:45,635 [275] [DEBUG] [app] Starting request: urn:request:90068272-6079-4265-8d48-b1eb275ecf77 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:45,636 [276] [DEBUG] [app] Starting request: urn:request:5056614d-61f8-45b4-a1d5-8576adde3d6c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:48:45,636 [275] [DEBUG] [app] Ending request: urn:request:90068272-6079-4265-8d48-b1eb275ecf77 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:90068272-6079-4265-8d48-b1eb275ecf77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:45,636 [276] [DEBUG] [app] Ending request: urn:request:5056614d-61f8-45b4-a1d5-8576adde3d6c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5056614d-61f8-45b4-a1d5-8576adde3d6c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:48:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:48:45,637 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:48:45,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:45,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:48:45,640 [282] [DEBUG] [app] Starting request: urn:request:42056239-3e76-4c99-8c56-f94b324e42db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:45,640 [285] [DEBUG] [app] Starting request: urn:request:931e2f6b-7b0e-4c8e-b6c2-3e2355f2b4d5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:48:45,641 [282] [DEBUG] [app] Ending request: urn:request:42056239-3e76-4c99-8c56-f94b324e42db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:42056239-3e76-4c99-8c56-f94b324e42db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:45,641 [285] [DEBUG] [app] Ending request: urn:request:931e2f6b-7b0e-4c8e-b6c2-3e2355f2b4d5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:931e2f6b-7b0e-4c8e-b6c2-3e2355f2b4d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:48:45,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:45,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:48:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:48:45,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:48:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:48:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:48:45,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:45,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:48:45,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:45,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:48:45,663 [282] [DEBUG] [app] Ending request: urn:request:27b1a75d-df60-44ca-a8a8-06bc3d7debbd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:27b1a75d-df60-44ca-a8a8-06bc3d7debbd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:45,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:48:45,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:48:45,665 [286] [DEBUG] [app] Ending request: urn:request:ed9d6377-3cc2-4bbc-a198-e96d6d694d3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed9d6377-3cc2-4bbc-a198-e96d6d694d3c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:48:45,665 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:48:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.035)
gunicorn-web stdout | 2024-04-22 03:48:45,666 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:48:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:48:45,666 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:48:45,958 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:48:46,358 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:48:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:48:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:14.067050+00:00 (in 27.000668 seconds)
securityworker stdout | 2024-04-22 03:48:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:17 UTC)" (scheduled at 2024-04-22 03:48:47.065745+00:00)
securityworker stdout | 2024-04-22 03:48:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:48:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:48:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:48:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:48:47,084 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:48:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:48:47,088 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:17 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:48:47,148 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
namespacegcworker stdout | 2024-04-22 03:48:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:48:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:10.248668+00:00 (in 23.002249 seconds)
namespacegcworker stdout | 2024-04-22 03:48:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:47 UTC)" (scheduled at 2024-04-22 03:48:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:48:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:48:47,248 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 47, 246931), True, datetime.datetime(2024, 4, 22, 3, 48, 47, 246931), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:48:47,263 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:48:47,263 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:48:47,263 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:48:48,133 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:48:49,028 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 03:48:49,546 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-secscan stdout | 2024-04-22 03:48:49,555 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:48:49,558 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:48:49,561 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
blobuploadcleanupworker stdout | 2024-04-22 03:48:50,713 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:48:51,001 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
namespacegcworker stdout | 2024-04-22 03:48:51,083 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:48:51,087 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:48:51,090 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:48:51,093 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:48:51,096 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:48:51,100 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:48:51,105 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:48:51,108 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:48:51,111 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
builder stdout | 2024-04-22 03:48:51,194 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-web stdout | 2024-04-22 03:48:51,253 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:48:51,256 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:48:51,265 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:48:51,268 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:48:51,270 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:48:51,969 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:48:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:48:54.433683+00:00 (in 2.002263 seconds)
notificationworker stdout | 2024-04-22 03:48:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:02 UTC)" (scheduled at 2024-04-22 03:48:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:48:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:48:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 52, 431716), True, datetime.datetime(2024, 4, 22, 3, 48, 52, 431716), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:48:52,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:48:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:48:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:48:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:48:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:18.047111+00:00 (in 25.000977 seconds)
exportactionlogsworker stdout | 2024-04-22 03:48:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:53 UTC)" (scheduled at 2024-04-22 03:48:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:48:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:48:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 48, 53, 46516), True, datetime.datetime(2024, 4, 22, 3, 48, 53, 46516), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:48:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:48:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:48:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:48:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:48:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:02.430973+00:00 (in 7.996785 seconds)
notificationworker stdout | 2024-04-22 03:48:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:54 UTC)" (scheduled at 2024-04-22 03:48:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:48:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:48:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:49:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:48:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:48:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:28.130973+00:00 (in 29.999446 seconds)
buildlogsarchiver stdout | 2024-04-22 03:48:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:28 UTC)" (scheduled at 2024-04-22 03:48:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:48:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 48, 58, 131923), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:48:58,148 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:48:58,148 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:48:58,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:49:00,632 [282] [DEBUG] [app] Starting request: urn:request:a76905c1-3b35-45b1-85c7-0b8087a867ee (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:00,632 [284] [DEBUG] [app] Starting request: urn:request:09f16075-4abd-4461-8b3c-c3338b423ae2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:00,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:49:00,635 [275] [DEBUG] [app] Starting request: urn:request:361d684e-a96f-4938-8ba8-baeabd1ac864 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:00,636 [281] [DEBUG] [app] Starting request: urn:request:cf977f60-8e4a-4187-92d9-7dd97874ac46 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:00,636 [275] [DEBUG] [app] Ending request: urn:request:361d684e-a96f-4938-8ba8-baeabd1ac864 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:361d684e-a96f-4938-8ba8-baeabd1ac864', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:49:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:49:00,636 [281] [DEBUG] [app] Ending request: urn:request:cf977f60-8e4a-4187-92d9-7dd97874ac46 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf977f60-8e4a-4187-92d9-7dd97874ac46', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:49:00,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:49:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 03:49:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:00,639 [284] [DEBUG] [app] Starting request: urn:request:f20cd622-629f-451d-beb7-641b1b4680ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:00,639 [285] [DEBUG] [app] Starting request: urn:request:cdc914c4-53b5-4e1a-ad75-56a6253b59c2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:00,640 [285] [DEBUG] [app] Ending request: urn:request:cdc914c4-53b5-4e1a-ad75-56a6253b59c2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cdc914c4-53b5-4e1a-ad75-56a6253b59c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:49:00,640 [284] [DEBUG] [app] Ending request: urn:request:f20cd622-629f-451d-beb7-641b1b4680ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f20cd622-629f-451d-beb7-641b1b4680ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:49:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:49:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:00,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:00,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:00,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:00,662 [284] [DEBUG] [app] Ending request: urn:request:09f16075-4abd-4461-8b3c-c3338b423ae2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09f16075-4abd-4461-8b3c-c3338b423ae2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:00,662 [282] [DEBUG] [app] Ending request: urn:request:a76905c1-3b35-45b1-85c7-0b8087a867ee (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a76905c1-3b35-45b1-85c7-0b8087a867ee', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:00,662 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:49:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:49:00,663 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:49:00,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.032)
gcworker stdout | 2024-04-22 03:49:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:49:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:31.842666+00:00 (in 29.999424 seconds)
gcworker stdout | 2024-04-22 03:49:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:31 UTC)" (scheduled at 2024-04-22 03:49:01.842666+00:00)
gcworker stdout | 2024-04-22 03:49:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548141843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:49:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:49:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:49:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:49:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:49:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:09.033206+00:00 (in 6.997084 seconds)
repositorygcworker stdout | 2024-04-22 03:49:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:02 UTC)" (scheduled at 2024-04-22 03:49:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:49:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:49:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:49:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:12.430973+00:00 (in 9.999537 seconds)
notificationworker stdout | 2024-04-22 03:49:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:12 UTC)" (scheduled at 2024-04-22 03:49:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:49:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:49:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 2, 431716), True, datetime.datetime(2024, 4, 22, 3, 49, 2, 431716), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:49:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:49:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:49:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:49:04,199 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:49:05,449 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:49:08,065 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:49:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:49:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:02.035613+00:00 (in 53.001891 seconds)
repositorygcworker stdout | 2024-04-22 03:49:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:09 UTC)" (scheduled at 2024-04-22 03:49:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:49:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:49:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 9, 33990), True, datetime.datetime(2024, 4, 22, 3, 49, 9, 33990), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:49:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:49:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:49:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:49:09,204 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:49:09,594 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:49:09,627 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:49:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:49:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:40.238173+00:00 (in 29.999427 seconds)
autopruneworker stdout | 2024-04-22 03:49:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:40 UTC)" (scheduled at 2024-04-22 03:49:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:49:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757690247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:49:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:49:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:47.245913+00:00 (in 36.996763 seconds)
namespacegcworker stdout | 2024-04-22 03:49:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:10 UTC)" (scheduled at 2024-04-22 03:49:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:49:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:49:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:49:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:49:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:49:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:49:10,279 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:49:10,862 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:49:11,009 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
servicekey stdout | 2024-04-22 03:49:12,413 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
notificationworker stdout | 2024-04-22 03:49:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:22.430973+00:00 (in 9.999566 seconds)
notificationworker stdout | 2024-04-22 03:49:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:22 UTC)" (scheduled at 2024-04-22 03:49:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:49:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:49:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 12, 431703), True, datetime.datetime(2024, 4, 22, 3, 49, 12, 431703), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:49:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:49:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:49:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:22 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:49:13,248 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:49:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:49:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:17.065745+00:00 (in 2.998120 seconds)
securityworker stdout | 2024-04-22 03:49:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:44 UTC)" (scheduled at 2024-04-22 03:49:14.067050+00:00)
securityworker stdout | 2024-04-22 03:49:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:49:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:49:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:49:14,070 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:14,070 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:49:14,070 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:49:14,074 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:49:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:49:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:49:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:49:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:49:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 44, 14, 76196), 1, 2])
securityworker stdout | 2024-04-22 03:49:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:49:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:49:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 44, 14, 76196), 1, 2])
securityworker stdout | 2024-04-22 03:49:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:49:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:49:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:49:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:49:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:49:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:49:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:49:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:49:15,632 [286] [DEBUG] [app] Starting request: urn:request:e6984ce4-57f5-4e00-8e78-b45d7fbdfb4c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:15,632 [282] [DEBUG] [app] Starting request: urn:request:b3577421-7566-4ca1-9a55-c6be986a7f6b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:49:15,636 [283] [DEBUG] [app] Starting request: urn:request:057ab65e-b794-435c-bfec-788276146e29 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:15,636 [275] [DEBUG] [app] Starting request: urn:request:5778f4b7-d926-45f4-bc4f-6ec1b29d7b6d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:15,636 [275] [DEBUG] [app] Ending request: urn:request:5778f4b7-d926-45f4-bc4f-6ec1b29d7b6d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5778f4b7-d926-45f4-bc4f-6ec1b29d7b6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:49:15,636 [283] [DEBUG] [app] Ending request: urn:request:057ab65e-b794-435c-bfec-788276146e29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:057ab65e-b794-435c-bfec-788276146e29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:49:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:49:15,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:15,639 [282] [DEBUG] [app] Starting request: urn:request:7bb7231a-db20-48e2-9f72-11f646990629 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:15,640 [282] [DEBUG] [app] Ending request: urn:request:7bb7231a-db20-48e2-9f72-11f646990629 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7bb7231a-db20-48e2-9f72-11f646990629', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:49:15,640 [284] [DEBUG] [app] Starting request: urn:request:7bfc5f49-2ae2-4e56-8583-b9fe7551ed70 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:15,641 [284] [DEBUG] [app] Ending request: urn:request:7bfc5f49-2ae2-4e56-8583-b9fe7551ed70 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7bfc5f49-2ae2-4e56-8583-b9fe7551ed70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:49:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:15,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:15,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:15,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:15,662 [282] [DEBUG] [app] Ending request: urn:request:b3577421-7566-4ca1-9a55-c6be986a7f6b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b3577421-7566-4ca1-9a55-c6be986a7f6b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:49:15,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:49:15,663 [286] [DEBUG] [app] Ending request: urn:request:e6984ce4-57f5-4e00-8e78-b45d7fbdfb4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6984ce4-57f5-4e00-8e78-b45d7fbdfb4c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:15,663 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:49:15,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:49:15,702 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:49:15,995 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:49:16,395 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:49:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:49:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:44.067050+00:00 (in 27.000802 seconds)
securityworker stdout | 2024-04-22 03:49:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:47 UTC)" (scheduled at 2024-04-22 03:49:17.065745+00:00)
securityworker stdout | 2024-04-22 03:49:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:49:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:49:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:49:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:49:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:49:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:49:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:49:17,185 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:49:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:49:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:53.045554+00:00 (in 34.997753 seconds)
exportactionlogsworker stdout | 2024-04-22 03:49:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:18 UTC)" (scheduled at 2024-04-22 03:49:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:49:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:49:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:49:18,170 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:49:19,068 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:49:19,564 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:49:19,567 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:49:19,570 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:49:19,581 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:49:20,750 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:49:21,010 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:49:21,097 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:49:21,100 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:49:21,104 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:49:21,108 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:49:21,111 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:49:21,115 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
namespacegcworker stdout | 2024-04-22 03:49:21,118 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-registry stdout | 2024-04-22 03:49:21,121 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:49:21,124 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
builder stdout | 2024-04-22 03:49:21,230 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-web stdout | 2024-04-22 03:49:21,261 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:49:21,263 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:49:21,275 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:49:21,278 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:49:21,280 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:49:22,003 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:49:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:32.430973+00:00 (in 9.999525 seconds)
notificationworker stdout | 2024-04-22 03:49:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:32 UTC)" (scheduled at 2024-04-22 03:49:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:49:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:49:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 22, 431767), True, datetime.datetime(2024, 4, 22, 3, 49, 22, 431767), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:49:22,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:49:22,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:49:22,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:49:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:49:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:24.738365+00:00 (in 59.999519 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:49:24,738 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:24 UTC)" (scheduled at 2024-04-22 03:49:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:49:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:49:24,756 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:49:24,756 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:49:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:49:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:58.130973+00:00 (in 29.999560 seconds)
buildlogsarchiver stdout | 2024-04-22 03:49:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:58 UTC)" (scheduled at 2024-04-22 03:49:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:49:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 49, 28, 131796), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:49:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:49:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:49:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:49:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:31.933721+00:00 (in 2.002238 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:29 UTC)" (scheduled at 2024-04-22 03:49:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 29, 931793), True, datetime.datetime(2024, 4, 22, 3, 49, 29, 931793), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:49:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:49:30,632 [282] [DEBUG] [app] Starting request: urn:request:767a925a-c2ff-44c0-b581-0496bc23f1c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:30,632 [285] [DEBUG] [app] Starting request: urn:request:0811e8ea-ab9b-4e33-93a3-30238f0271d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:49:30,635 [275] [DEBUG] [app] Starting request: urn:request:0b4f8925-5d4c-4da2-aeae-c885cf057deb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:30,636 [276] [DEBUG] [app] Starting request: urn:request:3f1a58d7-6b38-4e66-9c2c-46aa4e516fcc (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:30,636 [275] [DEBUG] [app] Ending request: urn:request:0b4f8925-5d4c-4da2-aeae-c885cf057deb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b4f8925-5d4c-4da2-aeae-c885cf057deb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:49:30,636 [276] [DEBUG] [app] Ending request: urn:request:3f1a58d7-6b38-4e66-9c2c-46aa4e516fcc (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f1a58d7-6b38-4e66-9c2c-46aa4e516fcc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:49:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:49:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:49:30,637 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:49:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:30,640 [282] [DEBUG] [app] Starting request: urn:request:7a6c0b4d-1c6f-4a83-ac94-ecff6a96563f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:30,640 [282] [DEBUG] [app] Ending request: urn:request:7a6c0b4d-1c6f-4a83-ac94-ecff6a96563f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7a6c0b4d-1c6f-4a83-ac94-ecff6a96563f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:49:30,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:30,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:49:30,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:30,642 [286] [DEBUG] [app] Starting request: urn:request:c9ba8fd9-cadc-48dc-8e2f-1b745eb97673 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:30,643 [286] [DEBUG] [app] Ending request: urn:request:c9ba8fd9-cadc-48dc-8e2f-1b745eb97673 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9ba8fd9-cadc-48dc-8e2f-1b745eb97673', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:49:30,643 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:30,643 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:49:30,644 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:30,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:30,652 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:30,662 [285] [DEBUG] [app] Ending request: urn:request:0811e8ea-ab9b-4e33-93a3-30238f0271d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0811e8ea-ab9b-4e33-93a3-30238f0271d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:30,662 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:49:30,663 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:49:30,664 [282] [DEBUG] [app] Ending request: urn:request:767a925a-c2ff-44c0-b581-0496bc23f1c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:767a925a-c2ff-44c0-b581-0496bc23f1c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:30,664 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:49:30,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:49:31,843 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:49:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:01.842666+00:00 (in 29.999260 seconds)
gcworker stdout | 2024-04-22 03:49:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:01 UTC)" (scheduled at 2024-04-22 03:49:31.842666+00:00)
gcworker stdout | 2024-04-22 03:49:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548171843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:49:31,861 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:49:31,861 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:49:31,861 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:49:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:49:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:29.931002+00:00 (in 57.996810 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:49:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:31 UTC)" (scheduled at 2024-04-22 03:49:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:49:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:49:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:49:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:42.430973+00:00 (in 9.999495 seconds)
notificationworker stdout | 2024-04-22 03:49:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:42 UTC)" (scheduled at 2024-04-22 03:49:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:49:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:49:32,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 32, 432053), True, datetime.datetime(2024, 4, 22, 3, 49, 32, 432053), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:49:32,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:49:32,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:49:32,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:49:34,237 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:49:35,486 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:49:38,101 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:49:39,241 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:49:39,631 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:49:39,663 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:49:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:49:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:10.238173+00:00 (in 29.999519 seconds)
autopruneworker stdout | 2024-04-22 03:49:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:10 UTC)" (scheduled at 2024-04-22 03:49:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:49:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757720247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:49:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:49:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:49:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:49:40,316 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:49:40,899 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:49:41,046 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:49:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:52.430973+00:00 (in 9.999489 seconds)
notificationworker stdout | 2024-04-22 03:49:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:52 UTC)" (scheduled at 2024-04-22 03:49:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:49:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:49:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 42, 431758), True, datetime.datetime(2024, 4, 22, 3, 49, 42, 431758), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:49:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:49:42,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:49:42,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:49:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:49:42,450 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:49:43,287 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:49:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:49:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:47.065745+00:00 (in 2.998179 seconds)
securityworker stdout | 2024-04-22 03:49:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:14 UTC)" (scheduled at 2024-04-22 03:49:44.067050+00:00)
securityworker stdout | 2024-04-22 03:49:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:49:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:49:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:49:44,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:49:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:49:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:49:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:49:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:49:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:49:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:49:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 44, 44, 77459), 1, 2])
securityworker stdout | 2024-04-22 03:49:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:49:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:49:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 44, 44, 77459), 1, 2])
securityworker stdout | 2024-04-22 03:49:44,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:49:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:49:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:49:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:49:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:49:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:49:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:49:44,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:49:44,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:49:45,632 [286] [DEBUG] [app] Starting request: urn:request:d0e41efc-1f24-415a-ba93-6dd64e4a3e84 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:45,632 [282] [DEBUG] [app] Starting request: urn:request:8febd8e7-b1a6-4c27-b153-3f974eb7f25e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:49:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:49:45,636 [275] [DEBUG] [app] Starting request: urn:request:72ea4737-deb3-4d25-85b8-6ab0b0ddbf00 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:45,636 [276] [DEBUG] [app] Starting request: urn:request:e03ed83f-9800-45ec-b93e-30110f44c123 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:49:45,636 [275] [DEBUG] [app] Ending request: urn:request:72ea4737-deb3-4d25-85b8-6ab0b0ddbf00 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:72ea4737-deb3-4d25-85b8-6ab0b0ddbf00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:49:45,636 [276] [DEBUG] [app] Ending request: urn:request:e03ed83f-9800-45ec-b93e-30110f44c123 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e03ed83f-9800-45ec-b93e-30110f44c123', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:49:45,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:49:45,637 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:49:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:49:45,639 [286] [DEBUG] [app] Starting request: urn:request:95fe104f-737b-48eb-a736-6041daf37ccf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:49:45,640 [286] [DEBUG] [app] Ending request: urn:request:95fe104f-737b-48eb-a736-6041daf37ccf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95fe104f-737b-48eb-a736-6041daf37ccf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:49:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:45,640 [285] [DEBUG] [app] Starting request: urn:request:afe33f26-6391-4e2f-83c0-6a131141f341 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:49:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:49:45,640 [285] [DEBUG] [app] Ending request: urn:request:afe33f26-6391-4e2f-83c0-6a131141f341 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:afe33f26-6391-4e2f-83c0-6a131141f341', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:49:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:45,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:49:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:49:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:49:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:49:45,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:49:45,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:49:45,662 [282] [DEBUG] [app] Ending request: urn:request:8febd8e7-b1a6-4c27-b153-3f974eb7f25e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8febd8e7-b1a6-4c27-b153-3f974eb7f25e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:45,662 [286] [DEBUG] [app] Ending request: urn:request:d0e41efc-1f24-415a-ba93-6dd64e4a3e84 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d0e41efc-1f24-415a-ba93-6dd64e4a3e84', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:49:45,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:49:45,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:49:45,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:49:45,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:49:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:49:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
expiredappspecifictokenworker stdout | 2024-04-22 03:49:45,738 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:49:46,032 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:49:46,433 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:49:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:49:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:14.067050+00:00 (in 27.000784 seconds)
securityworker stdout | 2024-04-22 03:49:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:17 UTC)" (scheduled at 2024-04-22 03:49:47.065745+00:00)
securityworker stdout | 2024-04-22 03:49:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:49:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:49:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:49:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:49:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:49:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:49:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:17 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:49:47,222 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
namespacegcworker stdout | 2024-04-22 03:49:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:49:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:10.248668+00:00 (in 23.002304 seconds)
namespacegcworker stdout | 2024-04-22 03:49:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:47 UTC)" (scheduled at 2024-04-22 03:49:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:49:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:49:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 47, 246750), True, datetime.datetime(2024, 4, 22, 3, 49, 47, 246750), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:49:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:49:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:49:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:47 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:49:48,206 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:49:49,106 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:49:49,575 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:49:49,579 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:49:49,582 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:49:49,618 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:49:50,787 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:49:51,023 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:49:51,106 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:49:51,110 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:49:51,115 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:49:51,117 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:49:51,121 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:49:51,124 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:49:51,129 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:49:51,134 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:49:51,153 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:49:51,268 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
gunicorn-web stdout | 2024-04-22 03:49:51,271 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:49:51,274 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:49:51,283 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:49:51,287 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:49:51,289 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
repositoryactioncounter stdout | 2024-04-22 03:49:52,040 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:49:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:49:54.433683+00:00 (in 2.002241 seconds)
notificationworker stdout | 2024-04-22 03:49:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:02 UTC)" (scheduled at 2024-04-22 03:49:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:49:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:49:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 52, 431926), True, datetime.datetime(2024, 4, 22, 3, 49, 52, 431926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:49:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:49:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:49:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:49:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:49:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:18.047111+00:00 (in 25.000932 seconds)
exportactionlogsworker stdout | 2024-04-22 03:49:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:53 UTC)" (scheduled at 2024-04-22 03:49:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:49:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:49:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 49, 53, 46614), True, datetime.datetime(2024, 4, 22, 3, 49, 53, 46614), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:49:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:49:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:49:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:49:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:49:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:02.430973+00:00 (in 7.996749 seconds)
notificationworker stdout | 2024-04-22 03:49:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:54 UTC)" (scheduled at 2024-04-22 03:49:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:49:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:49:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:50:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:49:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:49:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:28.130973+00:00 (in 29.999556 seconds)
buildlogsarchiver stdout | 2024-04-22 03:49:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:28 UTC)" (scheduled at 2024-04-22 03:49:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:49:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 49, 58, 131779), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:49:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:49:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:49:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:50:00,632 [282] [DEBUG] [app] Starting request: urn:request:9045f417-408a-4e09-802c-d388344ff3a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:00,632 [285] [DEBUG] [app] Starting request: urn:request:29ae55bf-c5ac-4023-9242-7a9b7433bace (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:50:00,636 [277] [DEBUG] [app] Starting request: urn:request:b44d11a7-af01-4cc2-b8ec-826acf8d59c9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:00,636 [275] [DEBUG] [app] Starting request: urn:request:31964a0d-5f98-45d0-b484-7ed61ce54a96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:00,636 [277] [DEBUG] [app] Ending request: urn:request:b44d11a7-af01-4cc2-b8ec-826acf8d59c9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b44d11a7-af01-4cc2-b8ec-826acf8d59c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:00,636 [275] [DEBUG] [app] Ending request: urn:request:31964a0d-5f98-45d0-b484-7ed61ce54a96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:31964a0d-5f98-45d0-b484-7ed61ce54a96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:50:00,637 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:50:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:00,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:00,640 [282] [DEBUG] [app] Starting request: urn:request:1c8b1efe-f867-457b-b21e-68a0131afc48 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:00,640 [282] [DEBUG] [app] Ending request: urn:request:1c8b1efe-f867-457b-b21e-68a0131afc48 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1c8b1efe-f867-457b-b21e-68a0131afc48', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:50:00,640 [286] [DEBUG] [app] Starting request: urn:request:fc894589-c831-47dc-b11d-ccfb0c3a6829 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:50:00,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:00,641 [286] [DEBUG] [app] Ending request: urn:request:fc894589-c831-47dc-b11d-ccfb0c3a6829 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc894589-c831-47dc-b11d-ccfb0c3a6829', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:50:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:50:00,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:50:00,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:50:00,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:00,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:00,664 [285] [DEBUG] [app] Ending request: urn:request:29ae55bf-c5ac-4023-9242-7a9b7433bace (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29ae55bf-c5ac-4023-9242-7a9b7433bace', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:00,664 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:50:00,664 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:50:00,664 [282] [DEBUG] [app] Ending request: urn:request:9045f417-408a-4e09-802c-d388344ff3a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9045f417-408a-4e09-802c-d388344ff3a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:50:00,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:50:00,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gcworker stdout | 2024-04-22 03:50:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:50:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:31.842666+00:00 (in 29.999451 seconds)
gcworker stdout | 2024-04-22 03:50:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:31 UTC)" (scheduled at 2024-04-22 03:50:01.842666+00:00)
gcworker stdout | 2024-04-22 03:50:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548201843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:50:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:50:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:50:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:50:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:50:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:09.033206+00:00 (in 6.997072 seconds)
repositorygcworker stdout | 2024-04-22 03:50:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:02 UTC)" (scheduled at 2024-04-22 03:50:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:50:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:50:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:50:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:12.430973+00:00 (in 9.999520 seconds)
notificationworker stdout | 2024-04-22 03:50:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:12 UTC)" (scheduled at 2024-04-22 03:50:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:50:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:50:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 2, 431714), True, datetime.datetime(2024, 4, 22, 3, 50, 2, 431714), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:50:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:50:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:50:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:50:04,275 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:50:05,523 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:50:08,138 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:50:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:50:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:02.035613+00:00 (in 53.001870 seconds)
repositorygcworker stdout | 2024-04-22 03:50:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:09 UTC)" (scheduled at 2024-04-22 03:50:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:50:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:50:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 9, 34018), True, datetime.datetime(2024, 4, 22, 3, 50, 9, 34018), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:50:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:50:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:50:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:50:09,278 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:50:09,667 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:50:09,700 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:50:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:50:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:40.238173+00:00 (in 29.999504 seconds)
autopruneworker stdout | 2024-04-22 03:50:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:40 UTC)" (scheduled at 2024-04-22 03:50:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:50:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757750247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:50:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:50:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:47.245913+00:00 (in 36.996789 seconds)
namespacegcworker stdout | 2024-04-22 03:50:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:10 UTC)" (scheduled at 2024-04-22 03:50:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:50:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:50:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:50:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:50:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:50:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:50:10,345 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:50:10,936 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:50:11,082 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:50:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:22.430973+00:00 (in 9.999540 seconds)
notificationworker stdout | 2024-04-22 03:50:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:22 UTC)" (scheduled at 2024-04-22 03:50:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:50:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:50:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 12, 431866), True, datetime.datetime(2024, 4, 22, 3, 50, 12, 431866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:50:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:50:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:50:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:50:12,486 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:50:13,324 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:50:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:50:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:17.065745+00:00 (in 2.998192 seconds)
securityworker stdout | 2024-04-22 03:50:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:44 UTC)" (scheduled at 2024-04-22 03:50:14.067050+00:00)
securityworker stdout | 2024-04-22 03:50:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:50:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:50:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:50:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:50:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:50:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:50:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:50:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:50:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:50:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:50:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 45, 14, 77049), 1, 2])
securityworker stdout | 2024-04-22 03:50:14,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:50:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:50:14,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 45, 14, 77049), 1, 2])
securityworker stdout | 2024-04-22 03:50:14,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:50:14,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:50:14,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:14,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:50:14,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:50:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:50:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:50:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:50:14,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:50:15,632 [285] [DEBUG] [app] Starting request: urn:request:7018c8d5-bce8-4fe8-9e21-eda2d59dfc86 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:15,632 [286] [DEBUG] [app] Starting request: urn:request:8704d393-d806-42dc-bc42-90b28a0bd5ab (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:50:15,635 [275] [DEBUG] [app] Starting request: urn:request:9fe096ad-faa0-4aa4-86a0-b9a1e8455150 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:15,635 [283] [DEBUG] [app] Starting request: urn:request:eb69d04e-486c-4ebf-9101-3d0a9c29b043 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:15,636 [275] [DEBUG] [app] Ending request: urn:request:9fe096ad-faa0-4aa4-86a0-b9a1e8455150 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9fe096ad-faa0-4aa4-86a0-b9a1e8455150', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:15,636 [283] [DEBUG] [app] Ending request: urn:request:eb69d04e-486c-4ebf-9101-3d0a9c29b043 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb69d04e-486c-4ebf-9101-3d0a9c29b043', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:50:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:50:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:15,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:15,639 [282] [DEBUG] [app] Starting request: urn:request:06dbe3ee-2c95-4df0-a11e-e5980720492b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:15,640 [282] [DEBUG] [app] Ending request: urn:request:06dbe3ee-2c95-4df0-a11e-e5980720492b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06dbe3ee-2c95-4df0-a11e-e5980720492b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:50:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:15,640 [284] [DEBUG] [app] Starting request: urn:request:05c2a587-da84-4a1c-9987-97d0765d4ecd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:15,641 [284] [DEBUG] [app] Ending request: urn:request:05c2a587-da84-4a1c-9987-97d0765d4ecd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:05c2a587-da84-4a1c-9987-97d0765d4ecd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:50:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:50:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:15,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:50:15,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:15,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:15,664 [285] [DEBUG] [app] Ending request: urn:request:7018c8d5-bce8-4fe8-9e21-eda2d59dfc86 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7018c8d5-bce8-4fe8-9e21-eda2d59dfc86', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:15,664 [286] [DEBUG] [app] Ending request: urn:request:8704d393-d806-42dc-bc42-90b28a0bd5ab (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8704d393-d806-42dc-bc42-90b28a0bd5ab', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:15,664 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:50:15,664 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:50:15,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:50:15,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:50:15,776 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:50:16,069 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:50:16,470 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:50:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:50:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:44.067050+00:00 (in 27.000730 seconds)
securityworker stdout | 2024-04-22 03:50:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:47 UTC)" (scheduled at 2024-04-22 03:50:17.065745+00:00)
securityworker stdout | 2024-04-22 03:50:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:50:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:50:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:50:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:50:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:50:17,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:50:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:50:17,259 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:50:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:50:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:53.045554+00:00 (in 34.997897 seconds)
exportactionlogsworker stdout | 2024-04-22 03:50:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:18 UTC)" (scheduled at 2024-04-22 03:50:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:50:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:50:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:50:18,243 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:50:19,143 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:50:19,586 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:50:19,589 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:50:19,592 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:50:19,629 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:50:20,813 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:50:21,036 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:50:21,115 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:50:21,121 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:50:21,123 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:50:21,127 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:50:21,132 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:50:21,136 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:50:21,142 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:50:21,145 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:50:21,189 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:50:21,278 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:50:21,281 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:50:21,290 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:50:21,294 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:50:21,297 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:50:21,303 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:50:22,077 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:50:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:32.430973+00:00 (in 9.999542 seconds)
notificationworker stdout | 2024-04-22 03:50:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:32 UTC)" (scheduled at 2024-04-22 03:50:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:50:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:50:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 22, 431882), True, datetime.datetime(2024, 4, 22, 3, 50, 22, 431882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:50:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:50:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:50:22,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:50:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:50:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:24.738365+00:00 (in 59.999520 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:50:24,738 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:24 UTC)" (scheduled at 2024-04-22 03:50:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:50:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:50:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:50:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:50:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:50:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:58.130973+00:00 (in 29.999522 seconds)
buildlogsarchiver stdout | 2024-04-22 03:50:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:58 UTC)" (scheduled at 2024-04-22 03:50:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:50:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 50, 28, 131874), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:50:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:50:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:50:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:50:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:31.933721+00:00 (in 2.002256 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:29 UTC)" (scheduled at 2024-04-22 03:50:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 29, 931762), True, datetime.datetime(2024, 4, 22, 3, 50, 29, 931762), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:50:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:50:30,632 [286] [DEBUG] [app] Starting request: urn:request:452440c6-038d-432a-9729-c6017f3ff18a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:30,632 [285] [DEBUG] [app] Starting request: urn:request:893aa60a-500f-4233-bbe8-608229169ce4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:50:30,636 [275] [DEBUG] [app] Starting request: urn:request:36aae960-96c5-4e1b-aeff-1f945ecb95b3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:30,636 [283] [DEBUG] [app] Starting request: urn:request:b60d9223-baa7-467a-85f2-087e03bd6f71 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:30,636 [275] [DEBUG] [app] Ending request: urn:request:36aae960-96c5-4e1b-aeff-1f945ecb95b3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36aae960-96c5-4e1b-aeff-1f945ecb95b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:30,636 [283] [DEBUG] [app] Ending request: urn:request:b60d9223-baa7-467a-85f2-087e03bd6f71 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b60d9223-baa7-467a-85f2-087e03bd6f71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:50:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:30,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:50:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:30,640 [285] [DEBUG] [app] Starting request: urn:request:bb7b03e9-3027-400b-ab2f-34d0cf606c11 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:30,640 [285] [DEBUG] [app] Ending request: urn:request:bb7b03e9-3027-400b-ab2f-34d0cf606c11 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb7b03e9-3027-400b-ab2f-34d0cf606c11', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:50:30,640 [282] [DEBUG] [app] Starting request: urn:request:26fc3cbc-572a-42f0-b9ae-bdfc31c27c7b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:50:30,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:30,641 [282] [DEBUG] [app] Ending request: urn:request:26fc3cbc-572a-42f0-b9ae-bdfc31c27c7b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26fc3cbc-572a-42f0-b9ae-bdfc31c27c7b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:50:30,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:50:30,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:30,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:50:30,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:50:30,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:30,654 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:30,663 [286] [DEBUG] [app] Ending request: urn:request:452440c6-038d-432a-9729-c6017f3ff18a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:452440c6-038d-432a-9729-c6017f3ff18a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:30,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:50:30,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:50:30,666 [285] [DEBUG] [app] Ending request: urn:request:893aa60a-500f-4233-bbe8-608229169ce4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:893aa60a-500f-4233-bbe8-608229169ce4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:30,666 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.036)
gunicorn-web stdout | 2024-04-22 03:50:30,667 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:50:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:50:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:01.842666+00:00 (in 29.999497 seconds)
gcworker stdout | 2024-04-22 03:50:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:01 UTC)" (scheduled at 2024-04-22 03:50:31.842666+00:00)
gcworker stdout | 2024-04-22 03:50:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548231843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:50:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:50:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:50:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:50:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:50:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:29.931002+00:00 (in 57.996824 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:50:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:31 UTC)" (scheduled at 2024-04-22 03:50:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:50:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:50:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:50:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:42.430973+00:00 (in 9.999555 seconds)
notificationworker stdout | 2024-04-22 03:50:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:42 UTC)" (scheduled at 2024-04-22 03:50:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:50:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:50:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 32, 431715), True, datetime.datetime(2024, 4, 22, 3, 50, 32, 431715), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:50:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:50:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:50:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:50:34,313 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:50:35,559 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:50:38,175 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:50:39,316 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:50:39,703 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:50:39,737 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:50:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:50:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:10.238173+00:00 (in 29.999493 seconds)
autopruneworker stdout | 2024-04-22 03:50:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:10 UTC)" (scheduled at 2024-04-22 03:50:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:50:40,259 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757780258, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:50:40,265 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:50:40,265 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:50:40,265 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:50:40,381 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:50:40,973 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:50:41,119 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:50:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:52.430973+00:00 (in 9.999476 seconds)
notificationworker stdout | 2024-04-22 03:50:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:52 UTC)" (scheduled at 2024-04-22 03:50:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:50:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:50:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 42, 431763), True, datetime.datetime(2024, 4, 22, 3, 50, 42, 431763), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:50:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:50:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:50:42,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:50:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:50:42,523 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:50:43,361 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:50:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:50:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:47.065745+00:00 (in 2.998157 seconds)
securityworker stdout | 2024-04-22 03:50:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:14 UTC)" (scheduled at 2024-04-22 03:50:44.067050+00:00)
securityworker stdout | 2024-04-22 03:50:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:50:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:50:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:50:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:50:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:50:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:50:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:50:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:50:44,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:50:44,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:50:44,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 45, 44, 77768), 1, 2])
securityworker stdout | 2024-04-22 03:50:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:50:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:50:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 45, 44, 77768), 1, 2])
securityworker stdout | 2024-04-22 03:50:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:50:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:50:44,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:44,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:50:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:50:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:50:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:50:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:50:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:50:44,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:50:45,632 [284] [DEBUG] [app] Starting request: urn:request:d60c2e04-aa92-494d-aa74-e5a6b85c9d44 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:45,632 [285] [DEBUG] [app] Starting request: urn:request:4ee73104-2793-466b-8a05-8414b2e479cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:50:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:50:45,636 [275] [DEBUG] [app] Starting request: urn:request:0915ac73-1368-4d46-8e04-a741bd4b97d8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:45,636 [278] [DEBUG] [app] Starting request: urn:request:7c5fd349-6316-48f1-a7c3-dd2eedd1f646 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:50:45,636 [275] [DEBUG] [app] Ending request: urn:request:0915ac73-1368-4d46-8e04-a741bd4b97d8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0915ac73-1368-4d46-8e04-a741bd4b97d8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:45,637 [278] [DEBUG] [app] Ending request: urn:request:7c5fd349-6316-48f1-a7c3-dd2eedd1f646 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c5fd349-6316-48f1-a7c3-dd2eedd1f646', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:50:45,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:45,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:50:45,637 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:45,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:45,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:50:45,640 [284] [DEBUG] [app] Starting request: urn:request:8a8ae0a3-7d1a-48b4-8d4c-fb78054ad603 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:45,640 [284] [DEBUG] [app] Ending request: urn:request:8a8ae0a3-7d1a-48b4-8d4c-fb78054ad603 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8a8ae0a3-7d1a-48b4-8d4c-fb78054ad603', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:50:45,640 [286] [DEBUG] [app] Starting request: urn:request:196fda77-7ed6-47d0-b0d5-502b1adf8680 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:50:45,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:45,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:45,641 [286] [DEBUG] [app] Ending request: urn:request:196fda77-7ed6-47d0-b0d5-502b1adf8680 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:196fda77-7ed6-47d0-b0d5-502b1adf8680', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:50:45,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:50:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:50:45,642 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:50:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:50:45,642 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:50:45,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:50:45,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:45,651 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:50:45,664 [284] [DEBUG] [app] Ending request: urn:request:d60c2e04-aa92-494d-aa74-e5a6b85c9d44 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d60c2e04-aa92-494d-aa74-e5a6b85c9d44', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:45,664 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:50:45,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:50:45,677 [285] [DEBUG] [app] Ending request: urn:request:4ee73104-2793-466b-8a05-8414b2e479cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ee73104-2793-466b-8a05-8414b2e479cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:50:45,677 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:50:45,677 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:50:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:50:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.046 118 0.045)
expiredappspecifictokenworker stdout | 2024-04-22 03:50:45,812 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:50:46,106 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:50:46,506 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:50:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:50:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:14.067050+00:00 (in 27.000670 seconds)
securityworker stdout | 2024-04-22 03:50:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:17 UTC)" (scheduled at 2024-04-22 03:50:47.065745+00:00)
securityworker stdout | 2024-04-22 03:50:47,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:50:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:50:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:50:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:50:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:50:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:50:47,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:50:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:50:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:10.248668+00:00 (in 23.002186 seconds)
namespacegcworker stdout | 2024-04-22 03:50:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:47 UTC)" (scheduled at 2024-04-22 03:50:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:50:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:50:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 47, 246757), True, datetime.datetime(2024, 4, 22, 3, 50, 47, 246757), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:50:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:50:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:50:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:50:47,296 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:50:48,280 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:50:49,182 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:50:49,595 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:50:49,598 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:50:49,601 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:50:49,666 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:50:50,850 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:50:51,048 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:50:51,129 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:50:51,132 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:50:51,136 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:50:51,138 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:50:51,142 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:50:51,147 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:50:51,155 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:50:51,157 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:50:51,224 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:50:51,286 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:50:51,289 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:50:51,299 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:50:51,303 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:50:51,305 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:50:51,338 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:50:52,114 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:50:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:50:54.433683+00:00 (in 2.002249 seconds)
notificationworker stdout | 2024-04-22 03:50:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:02 UTC)" (scheduled at 2024-04-22 03:50:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:50:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:50:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 52, 431855), True, datetime.datetime(2024, 4, 22, 3, 50, 52, 431855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:50:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:50:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:50:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:50:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:50:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:18.047111+00:00 (in 25.000960 seconds)
exportactionlogsworker stdout | 2024-04-22 03:50:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:53 UTC)" (scheduled at 2024-04-22 03:50:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:50:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:50:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 50, 53, 46476), True, datetime.datetime(2024, 4, 22, 3, 50, 53, 46476), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:50:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:50:53,063 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:50:53,063 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:50:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:50:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:02.430973+00:00 (in 7.996806 seconds)
notificationworker stdout | 2024-04-22 03:50:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:54 UTC)" (scheduled at 2024-04-22 03:50:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:50:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:50:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:51:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:50:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:50:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:28.130973+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2024-04-22 03:50:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:28 UTC)" (scheduled at 2024-04-22 03:50:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:50:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 50, 58, 131854), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:50:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:50:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:50:58,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:51:00,632 [286] [DEBUG] [app] Starting request: urn:request:1c9c5fd7-a1f9-43aa-a69f-982f3dca8bfe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:00,632 [282] [DEBUG] [app] Starting request: urn:request:d6767680-476a-4531-a31d-37aa59605470 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:51:00,637 [278] [DEBUG] [app] Starting request: urn:request:97692766-c5ad-4d1f-8b3a-f15ed6a55b12 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:00,637 [275] [DEBUG] [app] Starting request: urn:request:a29d926e-c303-4a93-a172-44d944f96e55 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:00,637 [278] [DEBUG] [app] Ending request: urn:request:97692766-c5ad-4d1f-8b3a-f15ed6a55b12 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97692766-c5ad-4d1f-8b3a-f15ed6a55b12', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:00,637 [275] [DEBUG] [app] Ending request: urn:request:a29d926e-c303-4a93-a172-44d944f96e55 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a29d926e-c303-4a93-a172-44d944f96e55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:00,637 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:51:00,638 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:00,638 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:00,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:00,640 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:00,640 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:00,641 [286] [DEBUG] [app] Starting request: urn:request:6db55d9c-a381-46c1-b135-04d74233f407 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:00,642 [285] [DEBUG] [app] Starting request: urn:request:8aedf659-a4f4-4960-9b05-0abfc078b575 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:00,642 [286] [DEBUG] [app] Ending request: urn:request:6db55d9c-a381-46c1-b135-04d74233f407 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6db55d9c-a381-46c1-b135-04d74233f407', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:00,642 [285] [DEBUG] [app] Ending request: urn:request:8aedf659-a4f4-4960-9b05-0abfc078b575 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8aedf659-a4f4-4960-9b05-0abfc078b575', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:00,642 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:00,642 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:51:00,642 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:00,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:00,643 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:00,643 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:00,652 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:00,653 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:00,664 [286] [DEBUG] [app] Ending request: urn:request:1c9c5fd7-a1f9-43aa-a69f-982f3dca8bfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1c9c5fd7-a1f9-43aa-a69f-982f3dca8bfe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:00,665 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:51:00,665 [282] [DEBUG] [app] Ending request: urn:request:d6767680-476a-4531-a31d-37aa59605470 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d6767680-476a-4531-a31d-37aa59605470', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:00,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:51:00,665 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 03:51:00,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gcworker stdout | 2024-04-22 03:51:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:51:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:31.842666+00:00 (in 29.999516 seconds)
gcworker stdout | 2024-04-22 03:51:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:31 UTC)" (scheduled at 2024-04-22 03:51:01.842666+00:00)
gcworker stdout | 2024-04-22 03:51:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548261843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:51:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:51:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:51:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:51:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:51:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:09.033206+00:00 (in 6.997043 seconds)
repositorygcworker stdout | 2024-04-22 03:51:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:02 UTC)" (scheduled at 2024-04-22 03:51:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:51:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:51:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:51:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:12.430973+00:00 (in 9.999528 seconds)
notificationworker stdout | 2024-04-22 03:51:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:12 UTC)" (scheduled at 2024-04-22 03:51:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:51:02,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:51:02,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 2, 432126), True, datetime.datetime(2024, 4, 22, 3, 51, 2, 432126), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:51:02,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:51:02,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:51:02,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:51:04,350 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:51:05,595 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:51:08,212 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:51:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:51:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:02.035613+00:00 (in 53.001954 seconds)
repositorygcworker stdout | 2024-04-22 03:51:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:09 UTC)" (scheduled at 2024-04-22 03:51:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:51:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:51:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 9, 33910), True, datetime.datetime(2024, 4, 22, 3, 51, 9, 33910), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:51:09,051 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:51:09,051 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:51:09,051 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:51:09,353 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:51:09,739 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:51:09,772 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:51:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:51:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:40.238173+00:00 (in 29.999471 seconds)
autopruneworker stdout | 2024-04-22 03:51:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:40 UTC)" (scheduled at 2024-04-22 03:51:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:51:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757810247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:51:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:51:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:30.247852+00:00 (in 19.998720 seconds)
namespacegcworker stdout | 2024-04-22 03:51:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:10 UTC)" (scheduled at 2024-04-22 03:51:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:51:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:51:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:51:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:51:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:51:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:51:10,390 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:51:11,009 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:51:11,154 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:51:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:22.430973+00:00 (in 9.999535 seconds)
notificationworker stdout | 2024-04-22 03:51:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:22 UTC)" (scheduled at 2024-04-22 03:51:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:51:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:51:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 12, 431847), True, datetime.datetime(2024, 4, 22, 3, 51, 12, 431847), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:51:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:51:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:51:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:51:12,560 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:51:13,397 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:51:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:51:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:17.065745+00:00 (in 2.998102 seconds)
securityworker stdout | 2024-04-22 03:51:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:44 UTC)" (scheduled at 2024-04-22 03:51:14.067050+00:00)
securityworker stdout | 2024-04-22 03:51:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:51:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:51:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:51:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:14,073 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:51:14,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:51:14,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:51:14,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:51:14,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:51:14,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:51:14,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:51:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 46, 14, 79180), 1, 2])
securityworker stdout | 2024-04-22 03:51:14,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:51:14,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:51:14,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 46, 14, 79180), 1, 2])
securityworker stdout | 2024-04-22 03:51:14,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:51:14,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:51:14,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:14,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:51:14,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:51:14,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:51:14,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:51:14,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:14,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:51:14,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:51:15,632 [282] [DEBUG] [app] Starting request: urn:request:7c3f46c3-afa4-483f-bc31-83f99c7a7b53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:15,632 [286] [DEBUG] [app] Starting request: urn:request:476f819a-e518-41f3-b947-baa639cb02cf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:51:15,636 [275] [DEBUG] [app] Starting request: urn:request:086f9988-6d8b-4464-99bf-ca4b0d3ce495 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:15,636 [283] [DEBUG] [app] Starting request: urn:request:6d26f692-9ff4-4d84-801b-e3205da10a4f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:15,636 [275] [DEBUG] [app] Ending request: urn:request:086f9988-6d8b-4464-99bf-ca4b0d3ce495 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:086f9988-6d8b-4464-99bf-ca4b0d3ce495', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:15,636 [283] [DEBUG] [app] Ending request: urn:request:6d26f692-9ff4-4d84-801b-e3205da10a4f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6d26f692-9ff4-4d84-801b-e3205da10a4f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:51:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:15,640 [285] [DEBUG] [app] Starting request: urn:request:7d11faee-17f5-4c92-b448-2129787a8c24 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:15,640 [285] [DEBUG] [app] Ending request: urn:request:7d11faee-17f5-4c92-b448-2129787a8c24 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d11faee-17f5-4c92-b448-2129787a8c24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:15,640 [284] [DEBUG] [app] Starting request: urn:request:6ec92a6d-d355-4707-952a-1eb26b220566 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:15,641 [284] [DEBUG] [app] Ending request: urn:request:6ec92a6d-d355-4707-952a-1eb26b220566 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ec92a6d-d355-4707-952a-1eb26b220566', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:15,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:51:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:15,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:15,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:15,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:15,662 [286] [DEBUG] [app] Ending request: urn:request:476f819a-e518-41f3-b947-baa639cb02cf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:476f819a-e518-41f3-b947-baa639cb02cf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:15,662 [282] [DEBUG] [app] Ending request: urn:request:7c3f46c3-afa4-483f-bc31-83f99c7a7b53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7c3f46c3-afa4-483f-bc31-83f99c7a7b53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:15,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:51:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:51:15,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:51:15,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
expiredappspecifictokenworker stdout | 2024-04-22 03:51:15,835 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:51:16,141 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:51:16,538 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:51:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:51:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:44.067050+00:00 (in 27.000789 seconds)
securityworker stdout | 2024-04-22 03:51:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:47 UTC)" (scheduled at 2024-04-22 03:51:17.065745+00:00)
securityworker stdout | 2024-04-22 03:51:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:51:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:51:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:51:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:51:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:51:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:51:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:51:17,318 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:51:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:51:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:53.045554+00:00 (in 34.997860 seconds)
exportactionlogsworker stdout | 2024-04-22 03:51:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:18 UTC)" (scheduled at 2024-04-22 03:51:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:51:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:51:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:51:18,316 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:51:19,215 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:51:19,603 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:51:19,605 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:51:19,607 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:51:19,690 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:51:20,886 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:51:21,057 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:51:21,137 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:51:21,142 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:51:21,145 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:51:21,149 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:51:21,152 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:51:21,158 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:51:21,164 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:51:21,166 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:51:21,235 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:51:21,293 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:51:21,296 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:51:21,307 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:51:21,310 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:51:21,312 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:51:21,374 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:51:22,148 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:51:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:32.430973+00:00 (in 9.999480 seconds)
notificationworker stdout | 2024-04-22 03:51:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:32 UTC)" (scheduled at 2024-04-22 03:51:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:51:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:51:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 22, 431977), True, datetime.datetime(2024, 4, 22, 3, 51, 22, 431977), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:51:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:51:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:51:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:51:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:51:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:24.738365+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:51:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:24 UTC)" (scheduled at 2024-04-22 03:51:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:51:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:51:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:51:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:51:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:51:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:58.130973+00:00 (in 29.999503 seconds)
buildlogsarchiver stdout | 2024-04-22 03:51:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:58 UTC)" (scheduled at 2024-04-22 03:51:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:51:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 51, 28, 131825), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:51:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:51:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:51:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:51:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:31.933721+00:00 (in 2.002176 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:29 UTC)" (scheduled at 2024-04-22 03:51:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,933 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 29, 931993), True, datetime.datetime(2024, 4, 22, 3, 51, 29, 931993), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,948 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,948 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:51:29,948 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:51:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:51:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:47.245913+00:00 (in 16.997503 seconds)
namespacegcworker stdout | 2024-04-22 03:51:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:56:30 UTC)" (scheduled at 2024-04-22 03:51:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 03:51:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 51, 30, 248750), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:51:30,263 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 51, 30, 248750), True, datetime.datetime(2024, 4, 22, 3, 51, 30, 248750), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:51:30,267 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 51, 30, 248750), True, datetime.datetime(2024, 4, 22, 3, 51, 30, 248750), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 3, 51, 30, 248750), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:51:30,271 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:51:30,271 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:56:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:51:30,632 [286] [DEBUG] [app] Starting request: urn:request:7513b0fa-e5a5-4cd6-af27-27be464d47e9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:30,632 [282] [DEBUG] [app] Starting request: urn:request:26fa5843-ea3f-43dd-8b63-44d5a8f0023a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:51:30,635 [275] [DEBUG] [app] Starting request: urn:request:5adbebef-5621-4b06-8665-08b6746dd67e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:30,635 [283] [DEBUG] [app] Starting request: urn:request:ecbad75d-c901-4b68-9431-0259e2dcca52 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:30,636 [275] [DEBUG] [app] Ending request: urn:request:5adbebef-5621-4b06-8665-08b6746dd67e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5adbebef-5621-4b06-8665-08b6746dd67e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:30,636 [283] [DEBUG] [app] Ending request: urn:request:ecbad75d-c901-4b68-9431-0259e2dcca52 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ecbad75d-c901-4b68-9431-0259e2dcca52', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:51:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:51:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:30,639 [286] [DEBUG] [app] Starting request: urn:request:83a92378-13f2-4f1e-a444-c1031fa40401 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:30,639 [286] [DEBUG] [app] Ending request: urn:request:83a92378-13f2-4f1e-a444-c1031fa40401 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83a92378-13f2-4f1e-a444-c1031fa40401', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:30,639 [285] [DEBUG] [app] Starting request: urn:request:22ab711d-2fca-4d36-acd4-8d2e797cfa56 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:51:30,639 [285] [DEBUG] [app] Ending request: urn:request:22ab711d-2fca-4d36-acd4-8d2e797cfa56 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:22ab711d-2fca-4d36-acd4-8d2e797cfa56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:51:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:30,661 [286] [DEBUG] [app] Ending request: urn:request:7513b0fa-e5a5-4cd6-af27-27be464d47e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7513b0fa-e5a5-4cd6-af27-27be464d47e9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:51:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:51:30,661 [282] [DEBUG] [app] Ending request: urn:request:26fa5843-ea3f-43dd-8b63-44d5a8f0023a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:26fa5843-ea3f-43dd-8b63-44d5a8f0023a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:51:30,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 03:51:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:51:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:01.842666+00:00 (in 29.999425 seconds)
gcworker stdout | 2024-04-22 03:51:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:01 UTC)" (scheduled at 2024-04-22 03:51:31.842666+00:00)
gcworker stdout | 2024-04-22 03:51:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548291843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:51:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:51:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:51:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:51:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:51:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:29.931002+00:00 (in 57.996768 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:51:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:31 UTC)" (scheduled at 2024-04-22 03:51:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:51:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:51:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:51:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:42.430973+00:00 (in 9.999499 seconds)
notificationworker stdout | 2024-04-22 03:51:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:42 UTC)" (scheduled at 2024-04-22 03:51:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:51:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:51:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 32, 431907), True, datetime.datetime(2024, 4, 22, 3, 51, 32, 431907), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:51:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:51:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:51:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:51:34,368 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:51:35,629 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:51:38,248 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:51:39,362 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:51:39,764 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:51:39,808 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:51:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:51:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:10.238173+00:00 (in 29.999426 seconds)
autopruneworker stdout | 2024-04-22 03:51:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:10 UTC)" (scheduled at 2024-04-22 03:51:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:51:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757840248, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:51:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:51:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:51:40,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:51:40,427 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:51:41,018 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:51:41,182 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:51:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:52.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 03:51:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:52 UTC)" (scheduled at 2024-04-22 03:51:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:51:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:51:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 42, 431972), True, datetime.datetime(2024, 4, 22, 3, 51, 42, 431972), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:51:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:51:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:51:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:51:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:51:42,597 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:51:43,419 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:51:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:51:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:47.065745+00:00 (in 2.998150 seconds)
securityworker stdout | 2024-04-22 03:51:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:14 UTC)" (scheduled at 2024-04-22 03:51:44.067050+00:00)
securityworker stdout | 2024-04-22 03:51:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:51:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:51:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:51:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:51:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:51:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:51:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:51:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:51:44,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:51:44,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:51:44,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 46, 44, 77938), 1, 2])
securityworker stdout | 2024-04-22 03:51:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:51:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:51:44,103 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 46, 44, 77938), 1, 2])
securityworker stdout | 2024-04-22 03:51:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:51:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:51:44,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:44,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:51:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:51:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:51:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:51:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:51:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:51:44,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:51:45,632 [282] [DEBUG] [app] Starting request: urn:request:d21119f7-b419-42d2-8eae-5e91376ae5f0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:45,632 [286] [DEBUG] [app] Starting request: urn:request:fbb6dced-21f2-461b-8f7b-288bccbd307f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:51:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:51:45,636 [281] [DEBUG] [app] Starting request: urn:request:35f12008-9713-4a36-a665-ee26a9f2acfd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:45,636 [283] [DEBUG] [app] Starting request: urn:request:02ddbcd4-1e5c-46cc-85f0-587849393498 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:51:45,636 [281] [DEBUG] [app] Ending request: urn:request:35f12008-9713-4a36-a665-ee26a9f2acfd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35f12008-9713-4a36-a665-ee26a9f2acfd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:45,636 [283] [DEBUG] [app] Ending request: urn:request:02ddbcd4-1e5c-46cc-85f0-587849393498 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:02ddbcd4-1e5c-46cc-85f0-587849393498', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:51:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:51:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:51:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:51:45,639 [286] [DEBUG] [app] Starting request: urn:request:2c134b8f-8662-402b-a524-2d2635b986af (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:45,639 [285] [DEBUG] [app] Starting request: urn:request:543cb1df-7da5-450e-a8fe-9d41998c8ce4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:51:45,639 [286] [DEBUG] [app] Ending request: urn:request:2c134b8f-8662-402b-a524-2d2635b986af (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2c134b8f-8662-402b-a524-2d2635b986af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:45,639 [285] [DEBUG] [app] Ending request: urn:request:543cb1df-7da5-450e-a8fe-9d41998c8ce4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:543cb1df-7da5-450e-a8fe-9d41998c8ce4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:51:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:51:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:51:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:51:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:51:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:51:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:51:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:51:45,660 [286] [DEBUG] [app] Ending request: urn:request:fbb6dced-21f2-461b-8f7b-288bccbd307f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fbb6dced-21f2-461b-8f7b-288bccbd307f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:51:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:51:45,661 [282] [DEBUG] [app] Ending request: urn:request:d21119f7-b419-42d2-8eae-5e91376ae5f0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d21119f7-b419-42d2-8eae-5e91376ae5f0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:51:45,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:51:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:51:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:51:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 03:51:45,873 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:51:46,162 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:51:46,573 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:51:47,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:51:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:14.067050+00:00 (in 27.000640 seconds)
securityworker stdout | 2024-04-22 03:51:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:17 UTC)" (scheduled at 2024-04-22 03:51:47.065745+00:00)
securityworker stdout | 2024-04-22 03:51:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:51:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:51:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:51:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:51:47,084 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:51:47,087 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:51:47,088 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:51:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:51:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:10.248668+00:00 (in 23.002238 seconds)
namespacegcworker stdout | 2024-04-22 03:51:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:47 UTC)" (scheduled at 2024-04-22 03:51:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:51:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:51:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 47, 246887), True, datetime.datetime(2024, 4, 22, 3, 51, 47, 246887), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:51:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:51:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:51:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:51:47,341 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:51:48,323 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:51:49,253 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:51:49,611 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:51:49,614 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:51:49,616 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:51:49,727 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:51:50,923 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:51:51,067 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:51:51,146 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:51:51,150 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:51:51,152 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:51:51,158 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:51:51,160 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:51:51,167 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:51:51,174 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:51:51,177 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:51:51,268 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:51:51,301 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:51:51,304 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:51:51,314 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:51:51,316 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:51:51,319 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:51:51,410 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:51:52,181 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:51:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:51:54.433683+00:00 (in 2.002209 seconds)
notificationworker stdout | 2024-04-22 03:51:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:02 UTC)" (scheduled at 2024-04-22 03:51:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:51:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:51:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 52, 431880), True, datetime.datetime(2024, 4, 22, 3, 51, 52, 431880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:51:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:51:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:51:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:51:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:51:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:18.047111+00:00 (in 25.000995 seconds)
exportactionlogsworker stdout | 2024-04-22 03:51:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:53 UTC)" (scheduled at 2024-04-22 03:51:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:51:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:51:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 51, 53, 46361), True, datetime.datetime(2024, 4, 22, 3, 51, 53, 46361), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:51:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:51:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:51:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:51:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:51:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:02.430973+00:00 (in 7.996725 seconds)
notificationworker stdout | 2024-04-22 03:51:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:54 UTC)" (scheduled at 2024-04-22 03:51:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:51:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:51:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:52:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:51:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:51:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:28.130973+00:00 (in 29.999500 seconds)
buildlogsarchiver stdout | 2024-04-22 03:51:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:28 UTC)" (scheduled at 2024-04-22 03:51:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:51:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 51, 58, 131873), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:51:58,148 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:51:58,148 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:51:58,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:52:00,632 [282] [DEBUG] [app] Starting request: urn:request:2db167d4-0511-4909-9549-20f31d64c341 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:00,632 [286] [DEBUG] [app] Starting request: urn:request:e2eaf1c3-5201-4d60-91d1-ce61382206e0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:52:00,635 [275] [DEBUG] [app] Starting request: urn:request:7154afa9-e289-47c3-bb84-77d925c6f8a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:00,636 [275] [DEBUG] [app] Ending request: urn:request:7154afa9-e289-47c3-bb84-77d925c6f8a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7154afa9-e289-47c3-bb84-77d925c6f8a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:00,636 [280] [DEBUG] [app] Starting request: urn:request:395979ad-5bc1-425d-85da-2f5a564631af (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:52:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:52:00,636 [280] [DEBUG] [app] Ending request: urn:request:395979ad-5bc1-425d-85da-2f5a564631af (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:395979ad-5bc1-425d-85da-2f5a564631af', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:52:00,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 03:52:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:00,639 [282] [DEBUG] [app] Starting request: urn:request:21df3abe-8ecb-40c6-b5cb-485bfe5a4d55 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:00,639 [282] [DEBUG] [app] Ending request: urn:request:21df3abe-8ecb-40c6-b5cb-485bfe5a4d55 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:21df3abe-8ecb-40c6-b5cb-485bfe5a4d55', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:00,639 [285] [DEBUG] [app] Starting request: urn:request:44d09690-7060-46ad-a57f-85192bc462e5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:00,640 [285] [DEBUG] [app] Ending request: urn:request:44d09690-7060-46ad-a57f-85192bc462e5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:44d09690-7060-46ad-a57f-85192bc462e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:52:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:52:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:00,661 [282] [DEBUG] [app] Ending request: urn:request:2db167d4-0511-4909-9549-20f31d64c341 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2db167d4-0511-4909-9549-20f31d64c341', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:52:00,661 [286] [DEBUG] [app] Ending request: urn:request:e2eaf1c3-5201-4d60-91d1-ce61382206e0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e2eaf1c3-5201-4d60-91d1-ce61382206e0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:52:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.031)
gunicorn-web stdout | 2024-04-22 03:52:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:52:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:52:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:31.842666+00:00 (in 29.999462 seconds)
gcworker stdout | 2024-04-22 03:52:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:31 UTC)" (scheduled at 2024-04-22 03:52:01.842666+00:00)
gcworker stdout | 2024-04-22 03:52:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548321843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:52:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:52:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:52:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:52:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:52:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:09.033206+00:00 (in 6.997057 seconds)
repositorygcworker stdout | 2024-04-22 03:52:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:02 UTC)" (scheduled at 2024-04-22 03:52:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:52:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:52:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:52:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:12.430973+00:00 (in 9.999538 seconds)
notificationworker stdout | 2024-04-22 03:52:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:12 UTC)" (scheduled at 2024-04-22 03:52:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:52:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:52:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 2, 431827), True, datetime.datetime(2024, 4, 22, 3, 52, 2, 431827), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:52:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:52:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:52:04,391 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:52:05,666 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:52:08,268 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:52:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:52:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:02.035613+00:00 (in 53.001896 seconds)
repositorygcworker stdout | 2024-04-22 03:52:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:09 UTC)" (scheduled at 2024-04-22 03:52:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:52:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:52:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 9, 34004), True, datetime.datetime(2024, 4, 22, 3, 52, 9, 34004), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:52:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:52:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:52:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:52:09,399 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:52:09,800 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:52:09,844 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:52:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:52:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:40.238173+00:00 (in 29.999426 seconds)
autopruneworker stdout | 2024-04-22 03:52:10,239 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:40 UTC)" (scheduled at 2024-04-22 03:52:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:52:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757870248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:52:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:52:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:47.245913+00:00 (in 36.996742 seconds)
namespacegcworker stdout | 2024-04-22 03:52:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:10 UTC)" (scheduled at 2024-04-22 03:52:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:52:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:52:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:52:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:52:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:52:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:52:10,445 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:52:11,048 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:52:11,196 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:52:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:22.430973+00:00 (in 9.999523 seconds)
notificationworker stdout | 2024-04-22 03:52:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:22 UTC)" (scheduled at 2024-04-22 03:52:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:52:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:52:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 12, 431853), True, datetime.datetime(2024, 4, 22, 3, 52, 12, 431853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:52:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:52:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:52:12,633 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:52:13,455 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:52:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:52:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:17.065745+00:00 (in 2.998038 seconds)
securityworker stdout | 2024-04-22 03:52:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:44 UTC)" (scheduled at 2024-04-22 03:52:14.067050+00:00)
securityworker stdout | 2024-04-22 03:52:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:52:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:52:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:52:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:52:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:52:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:52:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:52:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:52:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:52:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:52:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 47, 14, 78215), 1, 2])
securityworker stdout | 2024-04-22 03:52:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:52:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:52:14,105 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 47, 14, 78215), 1, 2])
securityworker stdout | 2024-04-22 03:52:14,109 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:52:14,109 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:52:14,109 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:14,109 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:52:14,109 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:52:14,109 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:52:14,109 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:52:14,111 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:14,111 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:52:14,111 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:52:15,632 [285] [DEBUG] [app] Starting request: urn:request:5da8ab09-b5b2-48dd-a749-5277c68e737f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:15,632 [282] [DEBUG] [app] Starting request: urn:request:cb97bd87-a981-47fe-a16f-9b336bd3d334 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:52:15,635 [275] [DEBUG] [app] Starting request: urn:request:c5469494-b361-4efe-9e1b-b5ca419328f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:15,635 [277] [DEBUG] [app] Starting request: urn:request:9c5ead8e-bae5-4fb4-bfb7-1cd35807331f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:15,636 [277] [DEBUG] [app] Ending request: urn:request:9c5ead8e-bae5-4fb4-bfb7-1cd35807331f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9c5ead8e-bae5-4fb4-bfb7-1cd35807331f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:15,636 [275] [DEBUG] [app] Ending request: urn:request:c5469494-b361-4efe-9e1b-b5ca419328f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5469494-b361-4efe-9e1b-b5ca419328f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:52:15,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:52:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:15,638 [285] [DEBUG] [app] Starting request: urn:request:6130b73f-3ea7-4537-b562-4e3d4ad00054 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:15,639 [285] [DEBUG] [app] Ending request: urn:request:6130b73f-3ea7-4537-b562-4e3d4ad00054 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6130b73f-3ea7-4537-b562-4e3d4ad00054', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:15,639 [286] [DEBUG] [app] Starting request: urn:request:9f3d905f-2461-4c4d-b93e-fff2467a0b10 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:52:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:15,639 [286] [DEBUG] [app] Ending request: urn:request:9f3d905f-2461-4c4d-b93e-fff2467a0b10 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f3d905f-2461-4c4d-b93e-fff2467a0b10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:52:15,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:15,660 [285] [DEBUG] [app] Ending request: urn:request:5da8ab09-b5b2-48dd-a749-5277c68e737f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5da8ab09-b5b2-48dd-a749-5277c68e737f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:52:15,660 [282] [DEBUG] [app] Ending request: urn:request:cb97bd87-a981-47fe-a16f-9b336bd3d334 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb97bd87-a981-47fe-a16f-9b336bd3d334', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:52:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:52:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 03:52:15,909 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:52:16,186 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:52:16,610 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:52:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:52:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:44.067050+00:00 (in 27.000729 seconds)
securityworker stdout | 2024-04-22 03:52:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:47 UTC)" (scheduled at 2024-04-22 03:52:17.065745+00:00)
securityworker stdout | 2024-04-22 03:52:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:52:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:52:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:52:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:52:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:52:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:52:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:52:17,378 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:52:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:52:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:23.046900+00:00 (in 4.999315 seconds)
exportactionlogsworker stdout | 2024-04-22 03:52:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:18 UTC)" (scheduled at 2024-04-22 03:52:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:52:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:52:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:52:18,359 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:52:19,290 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:52:19,619 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:52:19,622 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:52:19,625 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:52:19,754 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:52:20,959 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:52:21,077 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:52:21,157 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:52:21,159 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:52:21,164 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:52:21,167 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:52:21,171 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:52:21,175 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:52:21,183 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:52:21,187 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:52:21,278 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:52:21,309 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:52:21,312 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:52:21,322 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:52:21,324 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:52:21,327 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:52:21,445 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:52:22,218 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:52:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:32.430973+00:00 (in 9.999512 seconds)
notificationworker stdout | 2024-04-22 03:52:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:32 UTC)" (scheduled at 2024-04-22 03:52:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:52:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:52:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 22, 431885), True, datetime.datetime(2024, 4, 22, 3, 52, 22, 431885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:52:22,449 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:52:22,449 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:22,449 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:32 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:52:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:52:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:53.045554+00:00 (in 29.998081 seconds)
exportactionlogsworker stdout | 2024-04-22 03:52:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:57:23 UTC)" (scheduled at 2024-04-22 03:52:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 03:52:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 52, 23, 47701), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:52:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 52, 23, 47701), True, datetime.datetime(2024, 4, 22, 3, 52, 23, 47701), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:52:23,066 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 52, 23, 47701), True, datetime.datetime(2024, 4, 22, 3, 52, 23, 47701), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 3, 52, 23, 47701), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:52:23,069 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:52:23,070 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:57:23 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:52:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:52:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:24.738365+00:00 (in 59.999501 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:52:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:24 UTC)" (scheduled at 2024-04-22 03:52:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:52:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:52:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:52:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:52:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:52:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:58.130973+00:00 (in 29.999496 seconds)
buildlogsarchiver stdout | 2024-04-22 03:52:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:58 UTC)" (scheduled at 2024-04-22 03:52:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:52:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 52, 28, 131879), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:52:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:52:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:52:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:52:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:31.933721+00:00 (in 2.002194 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:29 UTC)" (scheduled at 2024-04-22 03:52:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 29, 931814), True, datetime.datetime(2024, 4, 22, 3, 52, 29, 931814), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:52:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:52:30,632 [286] [DEBUG] [app] Starting request: urn:request:0f10a642-1d02-4422-be9a-42bfcf895027 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:30,632 [282] [DEBUG] [app] Starting request: urn:request:c4ae0626-4360-4d8f-b43d-acbd0c1fad4c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:52:30,635 [283] [DEBUG] [app] Starting request: urn:request:2f3711cc-ddd4-4f44-8c7b-b5ed06ac0ac1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:30,635 [275] [DEBUG] [app] Starting request: urn:request:fb6a78ab-f614-43ca-8f41-f81369795f78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:30,635 [283] [DEBUG] [app] Ending request: urn:request:2f3711cc-ddd4-4f44-8c7b-b5ed06ac0ac1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2f3711cc-ddd4-4f44-8c7b-b5ed06ac0ac1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:30,635 [275] [DEBUG] [app] Ending request: urn:request:fb6a78ab-f614-43ca-8f41-f81369795f78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fb6a78ab-f614-43ca-8f41-f81369795f78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:52:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:52:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:30,638 [282] [DEBUG] [app] Starting request: urn:request:f497526f-0bf4-40b9-ac4d-5d622def8bf1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:30,639 [282] [DEBUG] [app] Ending request: urn:request:f497526f-0bf4-40b9-ac4d-5d622def8bf1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f497526f-0bf4-40b9-ac4d-5d622def8bf1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:30,639 [285] [DEBUG] [app] Starting request: urn:request:a4fa29aa-bccc-465f-8e70-f3a61e977141 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:52:30,639 [285] [DEBUG] [app] Ending request: urn:request:a4fa29aa-bccc-465f-8e70-f3a61e977141 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a4fa29aa-bccc-465f-8e70-f3a61e977141', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:52:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:30,660 [282] [DEBUG] [app] Ending request: urn:request:c4ae0626-4360-4d8f-b43d-acbd0c1fad4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4ae0626-4360-4d8f-b43d-acbd0c1fad4c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:52:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:52:30,661 [286] [DEBUG] [app] Ending request: urn:request:0f10a642-1d02-4422-be9a-42bfcf895027 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0f10a642-1d02-4422-be9a-42bfcf895027', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:52:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 03:52:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:52:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:01.842666+00:00 (in 29.999468 seconds)
gcworker stdout | 2024-04-22 03:52:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:01 UTC)" (scheduled at 2024-04-22 03:52:31.842666+00:00)
gcworker stdout | 2024-04-22 03:52:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548351843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:52:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:52:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:52:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:52:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:52:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:29.931002+00:00 (in 57.996824 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:52:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:31 UTC)" (scheduled at 2024-04-22 03:52:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:52:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:52:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:52:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:38.433394+00:00 (in 6.001910 seconds)
notificationworker stdout | 2024-04-22 03:52:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:42 UTC)" (scheduled at 2024-04-22 03:52:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:52:32,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:52:32,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 32, 432111), True, datetime.datetime(2024, 4, 22, 3, 52, 32, 432111), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:52:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:52:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:52:34,428 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:52:35,703 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:52:38,285 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
notificationworker stdout | 2024-04-22 03:52:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:42.430973+00:00 (in 3.997124 seconds)
notificationworker stdout | 2024-04-22 03:52:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:57:38 UTC)" (scheduled at 2024-04-22 03:52:38.433394+00:00)
notificationworker stdout | 2024-04-22 03:52:38,435 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 52, 38, 434255), 'notification/%'])
notificationworker stdout | 2024-04-22 03:52:38,448 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 52, 38, 434255), True, datetime.datetime(2024, 4, 22, 3, 52, 38, 434255), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 03:52:38,452 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 52, 38, 434255), True, datetime.datetime(2024, 4, 22, 3, 52, 38, 434255), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 3, 52, 38, 434255), 'notification/%'])
notificationworker stdout | 2024-04-22 03:52:38,456 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:38,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:57:38 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:52:39,427 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:52:39,821 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:52:39,875 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:52:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:52:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:10.238173+00:00 (in 29.999415 seconds)
autopruneworker stdout | 2024-04-22 03:52:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:10 UTC)" (scheduled at 2024-04-22 03:52:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:52:40,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757900248, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:52:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:52:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:52:40,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:52:40,481 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:52:41,073 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:52:41,233 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:52:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:52.430973+00:00 (in 9.999545 seconds)
notificationworker stdout | 2024-04-22 03:52:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:52 UTC)" (scheduled at 2024-04-22 03:52:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:52:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:52:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 42, 431864), True, datetime.datetime(2024, 4, 22, 3, 52, 42, 431864), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:52:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:52:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:52:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:52:42,669 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:52:43,492 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:52:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:52:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:47.065745+00:00 (in 2.998097 seconds)
securityworker stdout | 2024-04-22 03:52:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:14 UTC)" (scheduled at 2024-04-22 03:52:44.067050+00:00)
securityworker stdout | 2024-04-22 03:52:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:52:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:52:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:52:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:52:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:52:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:52:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:52:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:52:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:52:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:52:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 47, 44, 77926), 1, 2])
securityworker stdout | 2024-04-22 03:52:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:52:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:52:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 47, 44, 77926), 1, 2])
securityworker stdout | 2024-04-22 03:52:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:52:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:52:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:52:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:52:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:52:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:52:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:52:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:52:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:52:45,632 [282] [DEBUG] [app] Starting request: urn:request:2f300e6a-9872-4a4c-8c06-129f0ef0c020 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:45,632 [286] [DEBUG] [app] Starting request: urn:request:f4524405-597c-412d-a13f-9ed939cc80d2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:52:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:52:45,635 [283] [DEBUG] [app] Starting request: urn:request:522ecafe-fb35-4cf4-9fc2-e92570a9b437 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:45,636 [281] [DEBUG] [app] Starting request: urn:request:a0e804fa-9016-4d21-a053-b9e6d4312c98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:52:45,636 [283] [DEBUG] [app] Ending request: urn:request:522ecafe-fb35-4cf4-9fc2-e92570a9b437 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:522ecafe-fb35-4cf4-9fc2-e92570a9b437', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:45,636 [281] [DEBUG] [app] Ending request: urn:request:a0e804fa-9016-4d21-a053-b9e6d4312c98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a0e804fa-9016-4d21-a053-b9e6d4312c98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:52:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:52:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:52:45,639 [285] [DEBUG] [app] Starting request: urn:request:2bad8c1e-36aa-40a0-8839-b11cc65baffc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:45,639 [286] [DEBUG] [app] Starting request: urn:request:d788bc05-0a4a-4b3c-8d86-f99504adfb75 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:52:45,639 [285] [DEBUG] [app] Ending request: urn:request:2bad8c1e-36aa-40a0-8839-b11cc65baffc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2bad8c1e-36aa-40a0-8839-b11cc65baffc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:45,639 [286] [DEBUG] [app] Ending request: urn:request:d788bc05-0a4a-4b3c-8d86-f99504adfb75 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d788bc05-0a4a-4b3c-8d86-f99504adfb75', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:52:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:52:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:52:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:52:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:52:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:52:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:52:45,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:45,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:52:45,663 [282] [DEBUG] [app] Ending request: urn:request:2f300e6a-9872-4a4c-8c06-129f0ef0c020 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2f300e6a-9872-4a4c-8c06-129f0ef0c020', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:45,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:52:45,663 [286] [DEBUG] [app] Ending request: urn:request:f4524405-597c-412d-a13f-9ed939cc80d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f4524405-597c-412d-a13f-9ed939cc80d2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:52:45,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 03:52:45,664 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:52:45,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:52:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:52:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
expiredappspecifictokenworker stdout | 2024-04-22 03:52:45,947 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:52:46,202 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:52:46,646 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:52:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:52:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:14.067050+00:00 (in 27.000735 seconds)
securityworker stdout | 2024-04-22 03:52:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:17 UTC)" (scheduled at 2024-04-22 03:52:47.065745+00:00)
securityworker stdout | 2024-04-22 03:52:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:52:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:52:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:52:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:52:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:52:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:52:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:52:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:52:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:10.248668+00:00 (in 23.002240 seconds)
namespacegcworker stdout | 2024-04-22 03:52:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:47 UTC)" (scheduled at 2024-04-22 03:52:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:52:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:52:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 47, 246859), True, datetime.datetime(2024, 4, 22, 3, 52, 47, 246859), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:52:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:52:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:52:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:52:47,386 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:52:48,373 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:52:49,300 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:52:49,628 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:52:49,630 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:52:49,633 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:52:49,791 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:52:50,967 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:52:51,086 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:52:51,166 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:52:51,169 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:52:51,174 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:52:51,177 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:52:51,179 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:52:51,183 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:52:51,191 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:52:51,194 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
namespacegcworker stdout | 2024-04-22 03:52:51,293 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:52:51,316 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:52:51,320 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:52:51,329 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:52:51,332 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:52:51,334 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:52:51,477 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:52:52,228 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:52:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:52:54.433683+00:00 (in 2.002215 seconds)
notificationworker stdout | 2024-04-22 03:52:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:02 UTC)" (scheduled at 2024-04-22 03:52:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:52:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:52:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 52, 431868), True, datetime.datetime(2024, 4, 22, 3, 52, 52, 431868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:52:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:52:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:52:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:52:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:52:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:18.047111+00:00 (in 25.001040 seconds)
exportactionlogsworker stdout | 2024-04-22 03:52:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:53 UTC)" (scheduled at 2024-04-22 03:52:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:52:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:52:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 52, 53, 46393), True, datetime.datetime(2024, 4, 22, 3, 52, 53, 46393), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:52:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:52:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:52:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:52:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:52:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:02.430973+00:00 (in 7.996793 seconds)
notificationworker stdout | 2024-04-22 03:52:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:54 UTC)" (scheduled at 2024-04-22 03:52:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:52:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:52:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:53:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:52:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:52:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:28.130973+00:00 (in 29.999440 seconds)
buildlogsarchiver stdout | 2024-04-22 03:52:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:28 UTC)" (scheduled at 2024-04-22 03:52:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:52:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 52, 58, 131938), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:52:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:52:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:52:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:53:00,632 [285] [DEBUG] [app] Starting request: urn:request:9fce046f-4d68-4ce2-b05d-2eba219c31f7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:00,632 [286] [DEBUG] [app] Starting request: urn:request:019aef98-9b22-47ef-9d59-da12aa5b038f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:00,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:53:00,635 [275] [DEBUG] [app] Starting request: urn:request:86b71d83-6d9a-42a2-b6bd-ccc515b19b82 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:00,636 [275] [DEBUG] [app] Ending request: urn:request:86b71d83-6d9a-42a2-b6bd-ccc515b19b82 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:86b71d83-6d9a-42a2-b6bd-ccc515b19b82', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:53:00,636 [283] [DEBUG] [app] Starting request: urn:request:8ff5abb0-8694-4266-9b66-fe6ac4c8377c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:53:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:53:00,636 [283] [DEBUG] [app] Ending request: urn:request:8ff5abb0-8694-4266-9b66-fe6ac4c8377c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8ff5abb0-8694-4266-9b66-fe6ac4c8377c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:53:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:53:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:00,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:00,639 [282] [DEBUG] [app] Starting request: urn:request:dd17a322-dcfd-4027-80cc-86d9dbbbd56c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:00,639 [282] [DEBUG] [app] Ending request: urn:request:dd17a322-dcfd-4027-80cc-86d9dbbbd56c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dd17a322-dcfd-4027-80cc-86d9dbbbd56c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:00,639 [285] [DEBUG] [app] Starting request: urn:request:591c9a42-42b2-459b-9650-3632584fbdf9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:53:00,640 [285] [DEBUG] [app] Ending request: urn:request:591c9a42-42b2-459b-9650-3632584fbdf9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:591c9a42-42b2-459b-9650-3632584fbdf9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:53:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:00,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:00,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:00,662 [286] [DEBUG] [app] Ending request: urn:request:019aef98-9b22-47ef-9d59-da12aa5b038f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:019aef98-9b22-47ef-9d59-da12aa5b038f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:00,662 [285] [DEBUG] [app] Ending request: urn:request:9fce046f-4d68-4ce2-b05d-2eba219c31f7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9fce046f-4d68-4ce2-b05d-2eba219c31f7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:00,662 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 03:53:00,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 03:53:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:53:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:31.842666+00:00 (in 29.999508 seconds)
gcworker stdout | 2024-04-22 03:53:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:31 UTC)" (scheduled at 2024-04-22 03:53:01.842666+00:00)
gcworker stdout | 2024-04-22 03:53:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548381843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:53:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:53:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:53:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:53:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:53:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:09.033206+00:00 (in 6.997076 seconds)
repositorygcworker stdout | 2024-04-22 03:53:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:02 UTC)" (scheduled at 2024-04-22 03:53:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:53:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:53:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:53:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:12.430973+00:00 (in 9.999463 seconds)
notificationworker stdout | 2024-04-22 03:53:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:12 UTC)" (scheduled at 2024-04-22 03:53:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:53:02,432 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:53:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 2, 432096), True, datetime.datetime(2024, 4, 22, 3, 53, 2, 432096), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:53:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:53:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:53:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:53:04,444 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:53:05,739 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:53:08,322 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:53:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:53:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:26.035364+00:00 (in 17.001653 seconds)
repositorygcworker stdout | 2024-04-22 03:53:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:09 UTC)" (scheduled at 2024-04-22 03:53:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:53:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:53:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 9, 33974), True, datetime.datetime(2024, 4, 22, 3, 53, 9, 33974), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:53:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:53:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:53:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:53:09,453 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:53:09,832 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:53:09,886 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:53:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:53:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:40.238173+00:00 (in 29.999470 seconds)
autopruneworker stdout | 2024-04-22 03:53:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:40 UTC)" (scheduled at 2024-04-22 03:53:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:53:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757930247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:53:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:53:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:47.245913+00:00 (in 36.996733 seconds)
namespacegcworker stdout | 2024-04-22 03:53:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:10 UTC)" (scheduled at 2024-04-22 03:53:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:53:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:53:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:53:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:53:10,255 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:53:10,255 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:53:10,518 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:53:11,095 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:53:11,267 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:53:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:22.430973+00:00 (in 9.999471 seconds)
notificationworker stdout | 2024-04-22 03:53:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:22 UTC)" (scheduled at 2024-04-22 03:53:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:53:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:53:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 12, 431949), True, datetime.datetime(2024, 4, 22, 3, 53, 12, 431949), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:53:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:53:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:53:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:53:12,693 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:53:13,529 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:53:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:53:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:17.065745+00:00 (in 2.998061 seconds)
securityworker stdout | 2024-04-22 03:53:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:44 UTC)" (scheduled at 2024-04-22 03:53:14.067050+00:00)
securityworker stdout | 2024-04-22 03:53:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:53:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:53:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:53:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:53:14,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:53:14,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:53:14,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:53:14,093 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:53:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:53:14,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:53:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 48, 14, 78841), 1, 2])
securityworker stdout | 2024-04-22 03:53:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:53:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:53:14,103 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 48, 14, 78841), 1, 2])
securityworker stdout | 2024-04-22 03:53:14,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:53:14,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:53:14,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:14,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:53:14,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:53:14,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:53:14,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:53:14,109 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:14,109 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:53:14,109 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:53:15,632 [282] [DEBUG] [app] Starting request: urn:request:a0b84f87-7410-4f77-8001-07f9b1daa6a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:15,632 [284] [DEBUG] [app] Starting request: urn:request:8acece3f-71a1-42aa-831e-9fa6af142554 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:53:15,635 [279] [DEBUG] [app] Starting request: urn:request:1e1c1282-eab9-4e44-baca-ebab7ddaae3f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:15,636 [283] [DEBUG] [app] Starting request: urn:request:f0483f0a-4781-47d3-92f5-8bc9732d060f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:15,636 [279] [DEBUG] [app] Ending request: urn:request:1e1c1282-eab9-4e44-baca-ebab7ddaae3f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1e1c1282-eab9-4e44-baca-ebab7ddaae3f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:53:15,636 [283] [DEBUG] [app] Ending request: urn:request:f0483f0a-4781-47d3-92f5-8bc9732d060f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0483f0a-4781-47d3-92f5-8bc9732d060f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:53:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:53:15,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:53:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:53:15,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:15,639 [284] [DEBUG] [app] Starting request: urn:request:1535b563-f160-4a78-a1ab-df2b0d16b069 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:15,639 [284] [DEBUG] [app] Ending request: urn:request:1535b563-f160-4a78-a1ab-df2b0d16b069 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1535b563-f160-4a78-a1ab-df2b0d16b069', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:15,639 [286] [DEBUG] [app] Starting request: urn:request:164ed822-e0bc-458d-bc93-92177d90f9f0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:15,639 [286] [DEBUG] [app] Ending request: urn:request:164ed822-e0bc-458d-bc93-92177d90f9f0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:164ed822-e0bc-458d-bc93-92177d90f9f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:15,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:53:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:53:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:15,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:15,660 [284] [DEBUG] [app] Ending request: urn:request:8acece3f-71a1-42aa-831e-9fa6af142554 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8acece3f-71a1-42aa-831e-9fa6af142554', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:15,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:53:15,660 [282] [DEBUG] [app] Ending request: urn:request:a0b84f87-7410-4f77-8001-07f9b1daa6a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0b84f87-7410-4f77-8001-07f9b1daa6a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:53:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:53:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:53:15,976 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:53:16,236 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:53:16,683 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:53:17,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:53:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:44.067050+00:00 (in 27.000681 seconds)
securityworker stdout | 2024-04-22 03:53:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:47 UTC)" (scheduled at 2024-04-22 03:53:17.065745+00:00)
securityworker stdout | 2024-04-22 03:53:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:53:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:53:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:53:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:53:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:53:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:53:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:53:17,419 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:53:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:53:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:53.045554+00:00 (in 34.997930 seconds)
exportactionlogsworker stdout | 2024-04-22 03:53:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:18 UTC)" (scheduled at 2024-04-22 03:53:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:53:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:53:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:53:18,409 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:53:19,315 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:53:19,636 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:53:19,638 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:53:19,642 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:53:19,804 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:53:20,980 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:53:21,093 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:53:21,175 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:53:21,178 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:53:21,181 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:53:21,185 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:53:21,188 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:53:21,190 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:53:21,198 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:53:21,201 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:53:21,324 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:53:21,326 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
namespacegcworker stdout | 2024-04-22 03:53:21,329 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
gunicorn-web stdout | 2024-04-22 03:53:21,335 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:53:21,339 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:53:21,342 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
builder stdout | 2024-04-22 03:53:21,496 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:53:22,264 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:53:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:32.430973+00:00 (in 9.999532 seconds)
notificationworker stdout | 2024-04-22 03:53:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:32 UTC)" (scheduled at 2024-04-22 03:53:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:53:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:53:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 22, 431861), True, datetime.datetime(2024, 4, 22, 3, 53, 22, 431861), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:53:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:53:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:53:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:53:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:53:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:24.738365+00:00 (in 59.999484 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:53:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:24 UTC)" (scheduled at 2024-04-22 03:53:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:53:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:53:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:53:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:53:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:53:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:02.035613+00:00 (in 35.999806 seconds)
repositorygcworker stdout | 2024-04-22 03:53:26,035 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:58:26 UTC)" (scheduled at 2024-04-22 03:53:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 03:53:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 53, 26, 36032), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:53:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 53, 26, 36032), True, datetime.datetime(2024, 4, 22, 3, 53, 26, 36032), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:53:26,054 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 53, 26, 36032), True, datetime.datetime(2024, 4, 22, 3, 53, 26, 36032), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 3, 53, 26, 36032), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:53:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:53:26,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:58:26 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:53:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:53:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:58.130973+00:00 (in 29.999468 seconds)
buildlogsarchiver stdout | 2024-04-22 03:53:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:58 UTC)" (scheduled at 2024-04-22 03:53:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:53:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 53, 28, 131896), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:53:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:53:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:53:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:53:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:30.933445+00:00 (in 1.001955 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:29 UTC)" (scheduled at 2024-04-22 03:53:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 29, 931776), True, datetime.datetime(2024, 4, 22, 3, 53, 29, 931776), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:53:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:53:30,632 [284] [DEBUG] [app] Starting request: urn:request:097d6c21-328c-47b4-9de9-6fb000687a62 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:30,632 [282] [DEBUG] [app] Starting request: urn:request:d92c25f5-3580-4c3b-8b2b-83bac4b7619f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:53:30,636 [283] [DEBUG] [app] Starting request: urn:request:e78a9af1-8e44-443a-8767-ac49975016c0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:30,636 [280] [DEBUG] [app] Starting request: urn:request:0e28dc00-49c7-407b-b03d-b4d8812a7e72 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:30,636 [283] [DEBUG] [app] Ending request: urn:request:e78a9af1-8e44-443a-8767-ac49975016c0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e78a9af1-8e44-443a-8767-ac49975016c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:53:30,636 [280] [DEBUG] [app] Ending request: urn:request:0e28dc00-49c7-407b-b03d-b4d8812a7e72 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0e28dc00-49c7-407b-b03d-b4d8812a7e72', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:53:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:53:30,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:53:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:30,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:30,640 [286] [DEBUG] [app] Starting request: urn:request:f24ffeb6-7dee-48e0-bb07-dfd7622ac2b5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:30,640 [285] [DEBUG] [app] Starting request: urn:request:c2ded0e9-bd98-47ea-ab4e-3f513c77d431 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:30,640 [286] [DEBUG] [app] Ending request: urn:request:f24ffeb6-7dee-48e0-bb07-dfd7622ac2b5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f24ffeb6-7dee-48e0-bb07-dfd7622ac2b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:30,640 [285] [DEBUG] [app] Ending request: urn:request:c2ded0e9-bd98-47ea-ab4e-3f513c77d431 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2ded0e9-bd98-47ea-ab4e-3f513c77d431', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:53:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:53:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:53:30,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:53:30,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:30,661 [284] [DEBUG] [app] Ending request: urn:request:097d6c21-328c-47b4-9de9-6fb000687a62 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:097d6c21-328c-47b4-9de9-6fb000687a62', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:30,661 [282] [DEBUG] [app] Ending request: urn:request:d92c25f5-3580-4c3b-8b2b-83bac4b7619f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d92c25f5-3580-4c3b-8b2b-83bac4b7619f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:53:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:31.933721+00:00 (in 0.999822 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:58:30 UTC)" (scheduled at 2024-04-22 03:53:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 53, 30, 934144), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,947 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 53, 30, 934144), True, datetime.datetime(2024, 4, 22, 3, 53, 30, 934144), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,952 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 53, 30, 934144), True, datetime.datetime(2024, 4, 22, 3, 53, 30, 934144), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 3, 53, 30, 934144), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,955 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:53:30,956 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 03:58:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:53:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:53:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:01.842666+00:00 (in 29.999411 seconds)
gcworker stdout | 2024-04-22 03:53:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:01 UTC)" (scheduled at 2024-04-22 03:53:31.842666+00:00)
gcworker stdout | 2024-04-22 03:53:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548411843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:53:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:53:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:53:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:53:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:53:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:29.931002+00:00 (in 57.996804 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:53:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:31 UTC)" (scheduled at 2024-04-22 03:53:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:53:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:53:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:53:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:42.430973+00:00 (in 9.999571 seconds)
notificationworker stdout | 2024-04-22 03:53:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:42 UTC)" (scheduled at 2024-04-22 03:53:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:53:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:53:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 32, 431790), True, datetime.datetime(2024, 4, 22, 3, 53, 32, 431790), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:53:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:53:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:53:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:53:34,476 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:53:35,767 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:53:38,358 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:53:39,469 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:53:39,840 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:53:39,922 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:53:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:53:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:10.238173+00:00 (in 29.999495 seconds)
autopruneworker stdout | 2024-04-22 03:53:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:10 UTC)" (scheduled at 2024-04-22 03:53:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:53:40,254 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757960253, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:53:40,260 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:53:40,260 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:53:40,260 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:53:40,554 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:53:41,132 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:53:41,303 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:53:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:52.430973+00:00 (in 9.999534 seconds)
notificationworker stdout | 2024-04-22 03:53:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:52 UTC)" (scheduled at 2024-04-22 03:53:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:53:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:53:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 42, 431880), True, datetime.datetime(2024, 4, 22, 3, 53, 42, 431880), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:53:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:53:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:53:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:53:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:53:42,719 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:53:43,565 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:53:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:53:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:47.065745+00:00 (in 2.998075 seconds)
securityworker stdout | 2024-04-22 03:53:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:14 UTC)" (scheduled at 2024-04-22 03:53:44.067050+00:00)
securityworker stdout | 2024-04-22 03:53:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:53:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:53:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:53:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:53:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:53:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:53:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:53:44,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:53:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:53:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:53:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 48, 44, 78023), 1, 2])
securityworker stdout | 2024-04-22 03:53:44,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:53:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:53:44,103 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 48, 44, 78023), 1, 2])
securityworker stdout | 2024-04-22 03:53:44,107 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:53:44,107 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:53:44,107 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:44,107 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:53:44,107 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:53:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:53:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:53:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:53:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:53:44,109 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:53:45,632 [282] [DEBUG] [app] Starting request: urn:request:c31dd19b-90bd-4a59-9d26-b49f21565299 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:45,632 [285] [DEBUG] [app] Starting request: urn:request:d76d21d1-7cb2-440f-85f2-7aa2502b9464 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:53:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:53:45,635 [283] [DEBUG] [app] Starting request: urn:request:56066f55-fc4a-4dee-b236-6a8eca8b59f3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:45,635 [277] [DEBUG] [app] Starting request: urn:request:6dd8f6e7-a203-4c5a-a034-81f5e49c0263 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:53:45,635 [277] [DEBUG] [app] Ending request: urn:request:6dd8f6e7-a203-4c5a-a034-81f5e49c0263 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6dd8f6e7-a203-4c5a-a034-81f5e49c0263', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:53:45,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:53:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:53:45,636 [283] [DEBUG] [app] Ending request: urn:request:56066f55-fc4a-4dee-b236-6a8eca8b59f3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56066f55-fc4a-4dee-b236-6a8eca8b59f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:53:45,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.003 162 0.003)
gunicorn-web stdout | 2024-04-22 03:53:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:53:45,638 [285] [DEBUG] [app] Starting request: urn:request:311a68e5-449d-48f3-9a7f-f716fb770587 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:45,639 [285] [DEBUG] [app] Ending request: urn:request:311a68e5-449d-48f3-9a7f-f716fb770587 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:311a68e5-449d-48f3-9a7f-f716fb770587', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:53:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:53:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:45,639 [286] [DEBUG] [app] Starting request: urn:request:169fa885-04ef-4c75-b2da-2144ad1d146c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:53:45,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:45,640 [286] [DEBUG] [app] Ending request: urn:request:169fa885-04ef-4c75-b2da-2144ad1d146c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:169fa885-04ef-4c75-b2da-2144ad1d146c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:53:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:53:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:53:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:53:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:53:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:53:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:45,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:53:45,660 [285] [DEBUG] [app] Ending request: urn:request:d76d21d1-7cb2-440f-85f2-7aa2502b9464 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d76d21d1-7cb2-440f-85f2-7aa2502b9464', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:45,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:45,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:53:45,663 [282] [DEBUG] [app] Ending request: urn:request:c31dd19b-90bd-4a59-9d26-b49f21565299 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c31dd19b-90bd-4a59-9d26-b49f21565299', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:53:45,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:53:45,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:53:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:53:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
expiredappspecifictokenworker stdout | 2024-04-22 03:53:45,983 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:53:46,259 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:53:46,701 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:53:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:53:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:14.067050+00:00 (in 27.000752 seconds)
securityworker stdout | 2024-04-22 03:53:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:17 UTC)" (scheduled at 2024-04-22 03:53:47.065745+00:00)
securityworker stdout | 2024-04-22 03:53:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:53:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:53:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:53:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:53:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:53:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:53:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:53:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:53:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:10.248668+00:00 (in 23.002265 seconds)
namespacegcworker stdout | 2024-04-22 03:53:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:47 UTC)" (scheduled at 2024-04-22 03:53:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:53:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:53:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 47, 246846), True, datetime.datetime(2024, 4, 22, 3, 53, 47, 246846), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:53:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:53:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:53:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:53:47,456 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:53:48,433 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:53:49,351 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:53:49,644 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:53:49,647 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:53:49,650 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:53:49,840 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:53:50,989 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:53:51,106 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:53:51,184 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:53:51,188 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:53:51,190 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:53:51,194 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:53:51,197 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:53:51,202 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:53:51,205 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:53:51,208 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:53:51,332 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:53:51,335 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:53:51,342 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:53:51,347 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:53:51,350 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:53:51,363 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:53:51,528 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:53:52,300 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:53:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:53:54.433683+00:00 (in 2.002233 seconds)
notificationworker stdout | 2024-04-22 03:53:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:02 UTC)" (scheduled at 2024-04-22 03:53:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:53:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:53:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 52, 431860), True, datetime.datetime(2024, 4, 22, 3, 53, 52, 431860), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:53:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:53:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:53:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:53:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:53:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:18.047111+00:00 (in 25.001043 seconds)
exportactionlogsworker stdout | 2024-04-22 03:53:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:53 UTC)" (scheduled at 2024-04-22 03:53:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:53:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:53:53,048 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 53, 53, 46341), True, datetime.datetime(2024, 4, 22, 3, 53, 53, 46341), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:53:53,062 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:53:53,062 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:53:53,062 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:53:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:53:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:02.430973+00:00 (in 7.996772 seconds)
notificationworker stdout | 2024-04-22 03:53:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:54 UTC)" (scheduled at 2024-04-22 03:53:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:53:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:53:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:54:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:53:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:53:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:28.130973+00:00 (in 29.999482 seconds)
buildlogsarchiver stdout | 2024-04-22 03:53:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:28 UTC)" (scheduled at 2024-04-22 03:53:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:53:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 53, 58, 131899), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:53:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:53:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:53:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:54:00,632 [282] [DEBUG] [app] Starting request: urn:request:f3fcae56-0f38-492a-9e38-76296f7df944 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:00,632 [284] [DEBUG] [app] Starting request: urn:request:5706754d-9e2a-48bc-977c-037f0367839c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:54:00,636 [279] [DEBUG] [app] Starting request: urn:request:84f7740d-448f-4296-9834-936944ea12d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:00,636 [283] [DEBUG] [app] Starting request: urn:request:677394b2-27b1-488d-bf55-e2383b92ae96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:00,636 [283] [DEBUG] [app] Ending request: urn:request:677394b2-27b1-488d-bf55-e2383b92ae96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:677394b2-27b1-488d-bf55-e2383b92ae96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:00,636 [279] [DEBUG] [app] Ending request: urn:request:84f7740d-448f-4296-9834-936944ea12d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:84f7740d-448f-4296-9834-936944ea12d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:00,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:54:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:54:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:00,639 [282] [DEBUG] [app] Starting request: urn:request:59ab85c6-bf50-4ea4-b47d-cb3510eb3ae7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:00,639 [282] [DEBUG] [app] Ending request: urn:request:59ab85c6-bf50-4ea4-b47d-cb3510eb3ae7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59ab85c6-bf50-4ea4-b47d-cb3510eb3ae7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:54:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:00,640 [286] [DEBUG] [app] Starting request: urn:request:e81b44b5-07ef-4bfa-b10f-f6c8b3fd0274 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:00,640 [286] [DEBUG] [app] Ending request: urn:request:e81b44b5-07ef-4bfa-b10f-f6c8b3fd0274 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e81b44b5-07ef-4bfa-b10f-f6c8b3fd0274', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:54:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:54:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:00,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:00,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:00,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:00,661 [282] [DEBUG] [app] Ending request: urn:request:f3fcae56-0f38-492a-9e38-76296f7df944 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f3fcae56-0f38-492a-9e38-76296f7df944', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:54:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:54:00,661 [284] [DEBUG] [app] Ending request: urn:request:5706754d-9e2a-48bc-977c-037f0367839c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5706754d-9e2a-48bc-977c-037f0367839c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:54:00,662 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:54:00,662 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 03:54:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:54:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:31.842666+00:00 (in 29.999408 seconds)
gcworker stdout | 2024-04-22 03:54:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:31 UTC)" (scheduled at 2024-04-22 03:54:01.842666+00:00)
gcworker stdout | 2024-04-22 03:54:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548441843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:54:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:54:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:54:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:54:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:54:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:09.033206+00:00 (in 6.997136 seconds)
repositorygcworker stdout | 2024-04-22 03:54:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:02 UTC)" (scheduled at 2024-04-22 03:54:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:54:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:54:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:54:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:12.430973+00:00 (in 9.999536 seconds)
notificationworker stdout | 2024-04-22 03:54:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:12 UTC)" (scheduled at 2024-04-22 03:54:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:54:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:54:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 2, 431808), True, datetime.datetime(2024, 4, 22, 3, 54, 2, 431808), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:54:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:54:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:54:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:54:04,512 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:54:05,805 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:54:08,394 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:54:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:54:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:02.035613+00:00 (in 53.001901 seconds)
repositorygcworker stdout | 2024-04-22 03:54:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:09 UTC)" (scheduled at 2024-04-22 03:54:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:54:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:54:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 9, 33999), True, datetime.datetime(2024, 4, 22, 3, 54, 9, 33999), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:54:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:54:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:54:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:54:09,480 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:54:09,869 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:54:09,957 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:54:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:54:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:40.238173+00:00 (in 29.999476 seconds)
autopruneworker stdout | 2024-04-22 03:54:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:40 UTC)" (scheduled at 2024-04-22 03:54:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:54:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713757990247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:54:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:54:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:47.245913+00:00 (in 36.996775 seconds)
namespacegcworker stdout | 2024-04-22 03:54:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:10 UTC)" (scheduled at 2024-04-22 03:54:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:54:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:54:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:54:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:54:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:54:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:54:10,573 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:54:11,166 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:54:11,339 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:54:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:22.430973+00:00 (in 9.999516 seconds)
notificationworker stdout | 2024-04-22 03:54:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:22 UTC)" (scheduled at 2024-04-22 03:54:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:54:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:54:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 12, 431873), True, datetime.datetime(2024, 4, 22, 3, 54, 12, 431873), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:54:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:54:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:54:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:54:12,755 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:54:13,595 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:54:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:54:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:17.065745+00:00 (in 2.998165 seconds)
securityworker stdout | 2024-04-22 03:54:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:44 UTC)" (scheduled at 2024-04-22 03:54:14.067050+00:00)
securityworker stdout | 2024-04-22 03:54:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:54:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:54:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:54:14,070 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:54:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:54:14,074 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:54:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:54:14,088 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:54:14,092 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:54:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:54:14,093 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 49, 14, 75960), 1, 2])
securityworker stdout | 2024-04-22 03:54:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:54:14,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:54:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 49, 14, 75960), 1, 2])
securityworker stdout | 2024-04-22 03:54:14,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:54:14,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:54:14,101 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:14,101 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:54:14,101 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:54:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:54:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:54:14,103 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:14,103 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:54:14,103 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:54:15,632 [286] [DEBUG] [app] Starting request: urn:request:39adbb9a-f95b-4f79-a154-a2d4382c4cc6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:15,632 [282] [DEBUG] [app] Starting request: urn:request:ca2a9880-a019-492b-a6e2-759bc6e91258 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:15,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:54:15,636 [275] [DEBUG] [app] Starting request: urn:request:fcbd597a-e556-48f5-8763-034a0d7053a1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:15,636 [283] [DEBUG] [app] Starting request: urn:request:3f85ccba-48a4-460c-9cdc-ebe9be9113a3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:15,636 [275] [DEBUG] [app] Ending request: urn:request:fcbd597a-e556-48f5-8763-034a0d7053a1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fcbd597a-e556-48f5-8763-034a0d7053a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:15,637 [283] [DEBUG] [app] Ending request: urn:request:3f85ccba-48a4-460c-9cdc-ebe9be9113a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3f85ccba-48a4-460c-9cdc-ebe9be9113a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:54:15,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:54:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:15,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:15,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:15,640 [282] [DEBUG] [app] Starting request: urn:request:b4ab24d0-3d38-4ebf-ae83-719e7aca753c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:15,640 [284] [DEBUG] [app] Starting request: urn:request:2b717adf-a015-4f37-8014-56843d3a0eff (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:15,640 [282] [DEBUG] [app] Ending request: urn:request:b4ab24d0-3d38-4ebf-ae83-719e7aca753c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b4ab24d0-3d38-4ebf-ae83-719e7aca753c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:54:15,641 [284] [DEBUG] [app] Ending request: urn:request:2b717adf-a015-4f37-8014-56843d3a0eff (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2b717adf-a015-4f37-8014-56843d3a0eff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:54:15,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:54:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:15,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:54:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:15,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:15,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:15,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:15,662 [282] [DEBUG] [app] Ending request: urn:request:ca2a9880-a019-492b-a6e2-759bc6e91258 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ca2a9880-a019-492b-a6e2-759bc6e91258', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:54:15,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:54:15,663 [286] [DEBUG] [app] Ending request: urn:request:39adbb9a-f95b-4f79-a154-a2d4382c4cc6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:39adbb9a-f95b-4f79-a154-a2d4382c4cc6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:15,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:54:15,664 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.033)
expiredappspecifictokenworker stdout | 2024-04-22 03:54:15,990 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:54:16,286 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:54:16,737 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:54:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:54:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:44.067050+00:00 (in 27.000730 seconds)
securityworker stdout | 2024-04-22 03:54:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:47 UTC)" (scheduled at 2024-04-22 03:54:17.065745+00:00)
securityworker stdout | 2024-04-22 03:54:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:54:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:54:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:54:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:54:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:54:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:54:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:54:17,492 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:54:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:54:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:53.045554+00:00 (in 34.997920 seconds)
exportactionlogsworker stdout | 2024-04-22 03:54:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:18 UTC)" (scheduled at 2024-04-22 03:54:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:54:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:54:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:54:18,442 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:54:19,388 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:54:19,652 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:54:19,656 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:54:19,658 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:54:19,878 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:54:21,025 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:54:21,117 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:54:21,193 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:54:21,200 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:54:21,203 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:54:21,205 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:54:21,209 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:54:21,213 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:54:21,215 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:54:21,218 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:54:21,339 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:54:21,342 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:54:21,349 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:54:21,354 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:54:21,357 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:54:21,388 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:54:21,540 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:54:22,337 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:54:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:32.430973+00:00 (in 9.999525 seconds)
notificationworker stdout | 2024-04-22 03:54:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:32 UTC)" (scheduled at 2024-04-22 03:54:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:54:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:54:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 22, 431855), True, datetime.datetime(2024, 4, 22, 3, 54, 22, 431855), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:54:22,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:54:22,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:54:22,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:54:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:54:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:24.738365+00:00 (in 59.999469 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:54:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:24 UTC)" (scheduled at 2024-04-22 03:54:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:54:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:54:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:54:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:54:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:54:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:58.130973+00:00 (in 29.999516 seconds)
buildlogsarchiver stdout | 2024-04-22 03:54:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:58 UTC)" (scheduled at 2024-04-22 03:54:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:54:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 54, 28, 131864), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:54:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:54:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:54:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:54:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:31.933721+00:00 (in 2.002160 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:29 UTC)" (scheduled at 2024-04-22 03:54:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 29, 931864), True, datetime.datetime(2024, 4, 22, 3, 54, 29, 931864), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:54:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:54:30,632 [284] [DEBUG] [app] Starting request: urn:request:684cb602-290d-41da-8f7f-26fdc32470bb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:30,632 [286] [DEBUG] [app] Starting request: urn:request:e1a7571c-119e-4afb-b63e-44c073edff31 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:30,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:54:30,635 [281] [DEBUG] [app] Starting request: urn:request:e60c6ac1-1c65-4975-8671-1647c3dfde5a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:30,635 [275] [DEBUG] [app] Starting request: urn:request:99c939cc-591f-4375-a7ee-9d3a1527c931 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:30,635 [275] [DEBUG] [app] Ending request: urn:request:99c939cc-591f-4375-a7ee-9d3a1527c931 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99c939cc-591f-4375-a7ee-9d3a1527c931', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:30,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:54:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:54:30,636 [281] [DEBUG] [app] Ending request: urn:request:e60c6ac1-1c65-4975-8671-1647c3dfde5a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e60c6ac1-1c65-4975-8671-1647c3dfde5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.003)
gunicorn-registry stdout | 2024-04-22 03:54:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:30,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:30,638 [285] [DEBUG] [app] Starting request: urn:request:6edb4262-bce6-415b-910a-b171f3717fb6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:30,639 [285] [DEBUG] [app] Ending request: urn:request:6edb4262-bce6-415b-910a-b171f3717fb6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6edb4262-bce6-415b-910a-b171f3717fb6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:54:30,639 [286] [DEBUG] [app] Starting request: urn:request:823a9ff7-186f-4d28-bb98-c90bb93a1806 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:54:30,639 [286] [DEBUG] [app] Ending request: urn:request:823a9ff7-186f-4d28-bb98-c90bb93a1806 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:823a9ff7-186f-4d28-bb98-c90bb93a1806', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:54:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:30,660 [286] [DEBUG] [app] Ending request: urn:request:e1a7571c-119e-4afb-b63e-44c073edff31 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1a7571c-119e-4afb-b63e-44c073edff31', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:54:30,660 [284] [DEBUG] [app] Ending request: urn:request:684cb602-290d-41da-8f7f-26fdc32470bb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:684cb602-290d-41da-8f7f-26fdc32470bb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:30,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:54:30,660 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:54:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 03:54:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:54:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:01.842666+00:00 (in 29.999460 seconds)
gcworker stdout | 2024-04-22 03:54:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:01 UTC)" (scheduled at 2024-04-22 03:54:31.842666+00:00)
gcworker stdout | 2024-04-22 03:54:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548471843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:54:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:54:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:54:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:54:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:54:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:29.931002+00:00 (in 57.996812 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:54:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:31 UTC)" (scheduled at 2024-04-22 03:54:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:54:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:54:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:54:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:42.430973+00:00 (in 9.999537 seconds)
notificationworker stdout | 2024-04-22 03:54:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:42 UTC)" (scheduled at 2024-04-22 03:54:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:54:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:54:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 32, 431994), True, datetime.datetime(2024, 4, 22, 3, 54, 32, 431994), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:54:32,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:54:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:54:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:54:34,537 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:54:35,836 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:54:38,429 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:54:39,497 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:54:39,885 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:54:39,982 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:54:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:54:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:10.238173+00:00 (in 29.999501 seconds)
autopruneworker stdout | 2024-04-22 03:54:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:10 UTC)" (scheduled at 2024-04-22 03:54:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:54:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758020248, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:54:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:54:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:54:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:54:40,589 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:54:41,195 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:54:41,376 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:54:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:52.430973+00:00 (in 9.999499 seconds)
notificationworker stdout | 2024-04-22 03:54:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:52 UTC)" (scheduled at 2024-04-22 03:54:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:54:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:54:42,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 42, 431981), True, datetime.datetime(2024, 4, 22, 3, 54, 42, 431981), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:54:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:54:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:54:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:54:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:54:42,765 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:54:43,632 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:54:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:54:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:47.065745+00:00 (in 2.998143 seconds)
securityworker stdout | 2024-04-22 03:54:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:14 UTC)" (scheduled at 2024-04-22 03:54:44.067050+00:00)
securityworker stdout | 2024-04-22 03:54:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:54:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:54:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:54:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:54:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:54:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:54:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:54:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:54:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:54:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:54:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 49, 44, 77468), 1, 2])
securityworker stdout | 2024-04-22 03:54:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:54:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:54:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 49, 44, 77468), 1, 2])
securityworker stdout | 2024-04-22 03:54:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:54:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:54:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:54:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:54:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:54:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:54:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:54:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:54:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:54:45,632 [286] [DEBUG] [app] Starting request: urn:request:6a265a51-4f3e-4706-abbc-40951e7f4ded (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:45,632 [282] [DEBUG] [app] Starting request: urn:request:e302c1cf-b164-404a-8c6d-0dd4acde8ba3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:54:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:54:45,635 [281] [DEBUG] [app] Starting request: urn:request:721f75d9-c097-4624-b847-9c174cee926e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:45,635 [275] [DEBUG] [app] Starting request: urn:request:9ff6c6bc-3404-4796-8030-5ed9ccd2934f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:54:45,635 [281] [DEBUG] [app] Ending request: urn:request:721f75d9-c097-4624-b847-9c174cee926e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:721f75d9-c097-4624-b847-9c174cee926e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:45,635 [275] [DEBUG] [app] Ending request: urn:request:9ff6c6bc-3404-4796-8030-5ed9ccd2934f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9ff6c6bc-3404-4796-8030-5ed9ccd2934f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:54:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:54:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:54:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:54:45,638 [285] [DEBUG] [app] Starting request: urn:request:495f65d8-5a73-40fb-b813-0b1eb62a81ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:45,639 [285] [DEBUG] [app] Ending request: urn:request:495f65d8-5a73-40fb-b813-0b1eb62a81ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:495f65d8-5a73-40fb-b813-0b1eb62a81ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:54:45,639 [284] [DEBUG] [app] Starting request: urn:request:f56586ac-f250-49ad-b062-fa60576448c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:54:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:54:45,639 [284] [DEBUG] [app] Ending request: urn:request:f56586ac-f250-49ad-b062-fa60576448c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f56586ac-f250-49ad-b062-fa60576448c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:54:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:45,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:54:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:54:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:54:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:54:45,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:54:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:45,659 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:54:45,660 [286] [DEBUG] [app] Ending request: urn:request:6a265a51-4f3e-4706-abbc-40951e7f4ded (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6a265a51-4f3e-4706-abbc-40951e7f4ded', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:54:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:54:45,671 [282] [DEBUG] [app] Ending request: urn:request:e302c1cf-b164-404a-8c6d-0dd4acde8ba3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e302c1cf-b164-404a-8c6d-0dd4acde8ba3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:54:45,671 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:54:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.040 118 0.041)
gunicorn-web stdout | 2024-04-22 03:54:45,671 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:54:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:54:46,026 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:54:46,322 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:54:46,774 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:54:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:54:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:14.067050+00:00 (in 27.000760 seconds)
securityworker stdout | 2024-04-22 03:54:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:17 UTC)" (scheduled at 2024-04-22 03:54:47.065745+00:00)
securityworker stdout | 2024-04-22 03:54:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:54:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:54:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:54:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:54:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:54:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:54:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:54:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:54:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:10.248668+00:00 (in 23.002258 seconds)
namespacegcworker stdout | 2024-04-22 03:54:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:47 UTC)" (scheduled at 2024-04-22 03:54:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:54:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:54:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 47, 246840), True, datetime.datetime(2024, 4, 22, 3, 54, 47, 246840), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:54:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:54:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:54:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:54:47,527 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:54:48,458 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:54:49,403 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:54:49,660 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:54:49,663 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:54:49,668 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:54:49,910 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:54:51,062 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:54:51,129 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:54:51,206 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:54:51,213 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:54:51,218 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:54:51,221 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:54:51,223 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:54:51,225 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:54:51,228 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:54:51,230 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:54:51,346 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:54:51,350 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:54:51,358 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:54:51,361 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:54:51,365 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:54:51,404 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:54:51,576 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:54:52,373 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:54:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:54:54.433683+00:00 (in 2.002200 seconds)
notificationworker stdout | 2024-04-22 03:54:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:02 UTC)" (scheduled at 2024-04-22 03:54:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:54:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:54:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 52, 431999), True, datetime.datetime(2024, 4, 22, 3, 54, 52, 431999), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:54:52,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:54:52,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:54:52,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:54:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:54:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:18.047111+00:00 (in 25.001079 seconds)
exportactionlogsworker stdout | 2024-04-22 03:54:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:53 UTC)" (scheduled at 2024-04-22 03:54:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:54:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:54:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 54, 53, 46293), True, datetime.datetime(2024, 4, 22, 3, 54, 53, 46293), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:54:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:54:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:54:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:54:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:54:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:02.430973+00:00 (in 7.996763 seconds)
notificationworker stdout | 2024-04-22 03:54:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:54 UTC)" (scheduled at 2024-04-22 03:54:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:54:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:54:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:55:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:54:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:54:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:28.130973+00:00 (in 29.999419 seconds)
buildlogsarchiver stdout | 2024-04-22 03:54:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:28 UTC)" (scheduled at 2024-04-22 03:54:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:54:58,133 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 54, 58, 132017), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:54:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:54:58,148 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:54:58,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:55:00,633 [286] [DEBUG] [app] Starting request: urn:request:a6b6c7c4-71e2-4d36-bb1d-2088a6fa36a2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:00,633 [282] [DEBUG] [app] Starting request: urn:request:508a3a83-4149-456b-ab4c-1b32b6a1ef65 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:55:00,636 [279] [DEBUG] [app] Starting request: urn:request:35a88834-f75c-4c22-aa14-84b2479bf935 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:00,636 [275] [DEBUG] [app] Starting request: urn:request:a08d0e27-b66f-413e-9a8a-00c9dcfd68d1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:00,636 [275] [DEBUG] [app] Ending request: urn:request:a08d0e27-b66f-413e-9a8a-00c9dcfd68d1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a08d0e27-b66f-413e-9a8a-00c9dcfd68d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:55:00,636 [279] [DEBUG] [app] Ending request: urn:request:35a88834-f75c-4c22-aa14-84b2479bf935 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:35a88834-f75c-4c22-aa14-84b2479bf935', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:55:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:55:00,637 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:00,639 [286] [DEBUG] [app] Starting request: urn:request:bbe8eb99-36a2-47f1-9eec-2bb43db26fb6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:00,639 [285] [DEBUG] [app] Starting request: urn:request:f8d8916b-f83f-41f5-984d-7ca4ec00104c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:00,640 [286] [DEBUG] [app] Ending request: urn:request:bbe8eb99-36a2-47f1-9eec-2bb43db26fb6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bbe8eb99-36a2-47f1-9eec-2bb43db26fb6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:00,640 [285] [DEBUG] [app] Ending request: urn:request:f8d8916b-f83f-41f5-984d-7ca4ec00104c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f8d8916b-f83f-41f5-984d-7ca4ec00104c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:00,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:55:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:00,663 [282] [DEBUG] [app] Ending request: urn:request:508a3a83-4149-456b-ab4c-1b32b6a1ef65 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:508a3a83-4149-456b-ab4c-1b32b6a1ef65', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:00,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:55:00,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 03:55:00,671 [286] [DEBUG] [app] Ending request: urn:request:a6b6c7c4-71e2-4d36-bb1d-2088a6fa36a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6b6c7c4-71e2-4d36-bb1d-2088a6fa36a2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:00,671 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.040 118 0.040)
gunicorn-web stdout | 2024-04-22 03:55:00,672 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:55:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:55:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:31.842666+00:00 (in 29.999460 seconds)
gcworker stdout | 2024-04-22 03:55:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:31 UTC)" (scheduled at 2024-04-22 03:55:01.842666+00:00)
gcworker stdout | 2024-04-22 03:55:01,844 [67] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2024-04-22 03:55:01,858 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548501857, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:55:01,864 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:55:01,864 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:55:01,864 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:55:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:55:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:09.033206+00:00 (in 6.997071 seconds)
repositorygcworker stdout | 2024-04-22 03:55:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:02 UTC)" (scheduled at 2024-04-22 03:55:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:55:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:55:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:55:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:12.430973+00:00 (in 9.999471 seconds)
notificationworker stdout | 2024-04-22 03:55:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:12 UTC)" (scheduled at 2024-04-22 03:55:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:55:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:55:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 2, 431953), True, datetime.datetime(2024, 4, 22, 3, 55, 2, 431953), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:55:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:55:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:55:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:55:04,573 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:55:05,869 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:55:08,436 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:55:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:55:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:02.035613+00:00 (in 53.001935 seconds)
repositorygcworker stdout | 2024-04-22 03:55:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:09 UTC)" (scheduled at 2024-04-22 03:55:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:55:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:55:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 9, 34115), True, datetime.datetime(2024, 4, 22, 3, 55, 9, 34115), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:55:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:55:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:55:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:55:09,510 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:55:09,916 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:55:09,999 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:55:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:55:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:40.238173+00:00 (in 29.999468 seconds)
autopruneworker stdout | 2024-04-22 03:55:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:40 UTC)" (scheduled at 2024-04-22 03:55:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:55:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758050247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:55:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:55:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:47.245913+00:00 (in 36.996765 seconds)
namespacegcworker stdout | 2024-04-22 03:55:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:10 UTC)" (scheduled at 2024-04-22 03:55:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:55:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:55:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:55:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:55:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:55:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:55:10,625 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:55:11,231 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:55:11,413 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:55:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:22.430973+00:00 (in 9.999446 seconds)
notificationworker stdout | 2024-04-22 03:55:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:22 UTC)" (scheduled at 2024-04-22 03:55:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:55:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:55:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 12, 431980), True, datetime.datetime(2024, 4, 22, 3, 55, 12, 431980), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:55:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:55:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:55:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:55:12,781 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:55:13,667 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:55:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:55:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:17.065745+00:00 (in 2.998080 seconds)
securityworker stdout | 2024-04-22 03:55:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:44 UTC)" (scheduled at 2024-04-22 03:55:14.067050+00:00)
securityworker stdout | 2024-04-22 03:55:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:55:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:55:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:55:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:55:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:55:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:55:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:55:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:55:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:55:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:55:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 50, 14, 77853), 1, 2])
securityworker stdout | 2024-04-22 03:55:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:55:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:55:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 50, 14, 77853), 1, 2])
securityworker stdout | 2024-04-22 03:55:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:55:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:55:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:55:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:55:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:55:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:55:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:55:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:55:15,632 [285] [DEBUG] [app] Starting request: urn:request:ad758c6c-954e-46c7-9e40-c8db3da92b8f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:15,632 [284] [DEBUG] [app] Starting request: urn:request:ad88fa0c-a47f-42bf-94b4-0f8faf282bb7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:55:15,635 [283] [DEBUG] [app] Starting request: urn:request:40c312ad-a893-4725-9899-c027ae261d16 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:15,636 [283] [DEBUG] [app] Ending request: urn:request:40c312ad-a893-4725-9899-c027ae261d16 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:40c312ad-a893-4725-9899-c027ae261d16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:55:15,636 [281] [DEBUG] [app] Starting request: urn:request:e564e185-8239-466f-8f23-b899d645decf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:55:15,636 [281] [DEBUG] [app] Ending request: urn:request:e564e185-8239-466f-8f23-b899d645decf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e564e185-8239-466f-8f23-b899d645decf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:55:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:55:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:15,639 [286] [DEBUG] [app] Starting request: urn:request:3ce5f6e6-0ab8-46fc-9dec-6ed1263a8645 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:15,639 [286] [DEBUG] [app] Ending request: urn:request:3ce5f6e6-0ab8-46fc-9dec-6ed1263a8645 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ce5f6e6-0ab8-46fc-9dec-6ed1263a8645', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:15,639 [282] [DEBUG] [app] Starting request: urn:request:0329fabc-339e-4319-ac6f-887892a72921 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:55:15,640 [282] [DEBUG] [app] Ending request: urn:request:0329fabc-339e-4319-ac6f-887892a72921 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0329fabc-339e-4319-ac6f-887892a72921', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:55:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:15,660 [284] [DEBUG] [app] Ending request: urn:request:ad88fa0c-a47f-42bf-94b4-0f8faf282bb7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ad88fa0c-a47f-42bf-94b4-0f8faf282bb7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:55:15,660 [285] [DEBUG] [app] Ending request: urn:request:ad758c6c-954e-46c7-9e40-c8db3da92b8f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ad758c6c-954e-46c7-9e40-c8db3da92b8f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:55:15,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.028)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.028)
gunicorn-web stdout | 2024-04-22 03:55:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:55:16,040 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:55:16,358 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:55:16,810 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:55:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:55:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:44.067050+00:00 (in 27.000777 seconds)
securityworker stdout | 2024-04-22 03:55:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:47 UTC)" (scheduled at 2024-04-22 03:55:17.065745+00:00)
securityworker stdout | 2024-04-22 03:55:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:55:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:55:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:55:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:55:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:55:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:55:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:55:17,564 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:55:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:55:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:53.045554+00:00 (in 34.997930 seconds)
exportactionlogsworker stdout | 2024-04-22 03:55:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:18 UTC)" (scheduled at 2024-04-22 03:55:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:55:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:55:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:55:18,469 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:55:19,432 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:55:19,669 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:55:19,673 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:55:19,675 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:55:19,937 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:55:21,099 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:55:21,140 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:55:21,218 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:55:21,221 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:55:21,231 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:55:21,238 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:55:21,242 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:55:21,246 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:55:21,251 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:55:21,255 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:55:21,355 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:55:21,356 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:55:21,365 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:55:21,368 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:55:21,373 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:55:21,420 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:55:21,614 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 03:55:22,410 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 03:55:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:32.430973+00:00 (in 9.999448 seconds)
notificationworker stdout | 2024-04-22 03:55:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:32 UTC)" (scheduled at 2024-04-22 03:55:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:55:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:55:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 22, 431946), True, datetime.datetime(2024, 4, 22, 3, 55, 22, 431946), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:55:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:55:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:55:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:55:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:55:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:24.738365+00:00 (in 59.999505 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:55:24,738 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:24 UTC)" (scheduled at 2024-04-22 03:55:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:55:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:55:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:55:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:55:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:55:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:58.130973+00:00 (in 29.999464 seconds)
buildlogsarchiver stdout | 2024-04-22 03:55:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:58 UTC)" (scheduled at 2024-04-22 03:55:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:55:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 55, 28, 131782), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:55:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:55:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:55:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:55:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:31.933721+00:00 (in 2.002217 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:29 UTC)" (scheduled at 2024-04-22 03:55:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 29, 931761), True, datetime.datetime(2024, 4, 22, 3, 55, 29, 931761), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,945 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,945 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:55:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:55:30,632 [285] [DEBUG] [app] Starting request: urn:request:18e7885e-df11-485f-9409-301fb716a6c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:30,632 [282] [DEBUG] [app] Starting request: urn:request:b73cacdf-15bb-4e1a-8e0b-2813abf680a4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:55:30,635 [281] [DEBUG] [app] Starting request: urn:request:f4296a53-8b48-488c-92bc-ab037231696e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:30,635 [275] [DEBUG] [app] Starting request: urn:request:99fc5e95-a830-4f64-95ee-dae8c689783e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:30,635 [281] [DEBUG] [app] Ending request: urn:request:f4296a53-8b48-488c-92bc-ab037231696e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4296a53-8b48-488c-92bc-ab037231696e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:55:30,635 [275] [DEBUG] [app] Ending request: urn:request:99fc5e95-a830-4f64-95ee-dae8c689783e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:99fc5e95-a830-4f64-95ee-dae8c689783e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:55:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:55:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:55:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:30,638 [282] [DEBUG] [app] Starting request: urn:request:e0c07a9e-97c3-4fab-bd6b-d48d0d4f6827 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:30,639 [284] [DEBUG] [app] Starting request: urn:request:f7adf261-f657-4670-9ddb-9b219bcb5f8f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:30,639 [282] [DEBUG] [app] Ending request: urn:request:e0c07a9e-97c3-4fab-bd6b-d48d0d4f6827 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e0c07a9e-97c3-4fab-bd6b-d48d0d4f6827', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:30,639 [284] [DEBUG] [app] Ending request: urn:request:f7adf261-f657-4670-9ddb-9b219bcb5f8f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7adf261-f657-4670-9ddb-9b219bcb5f8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:55:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:55:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:30,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:30,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:30,660 [285] [DEBUG] [app] Ending request: urn:request:18e7885e-df11-485f-9409-301fb716a6c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:18e7885e-df11-485f-9409-301fb716a6c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:30,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:55:30,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:55:30,662 [282] [DEBUG] [app] Ending request: urn:request:b73cacdf-15bb-4e1a-8e0b-2813abf680a4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b73cacdf-15bb-4e1a-8e0b-2813abf680a4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:30,663 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:55:30,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gcworker stdout | 2024-04-22 03:55:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:55:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:01.842666+00:00 (in 29.999411 seconds)
gcworker stdout | 2024-04-22 03:55:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:01 UTC)" (scheduled at 2024-04-22 03:55:31.842666+00:00)
gcworker stdout | 2024-04-22 03:55:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548531843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:55:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:55:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:55:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:55:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:55:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:29.931002+00:00 (in 57.996819 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:55:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:31 UTC)" (scheduled at 2024-04-22 03:55:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:55:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:55:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:55:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:42.430973+00:00 (in 9.999553 seconds)
notificationworker stdout | 2024-04-22 03:55:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:42 UTC)" (scheduled at 2024-04-22 03:55:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:55:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:55:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 32, 431910), True, datetime.datetime(2024, 4, 22, 3, 55, 32, 431910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:55:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:55:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:55:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:55:34,584 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:55:35,877 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:55:38,473 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:55:39,533 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:55:39,925 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:55:40,034 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:55:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:55:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:10.238173+00:00 (in 29.999468 seconds)
autopruneworker stdout | 2024-04-22 03:55:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:10 UTC)" (scheduled at 2024-04-22 03:55:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:55:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758080247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:55:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:55:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:55:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:55:40,642 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:55:41,261 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:55:41,446 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:55:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:52.430973+00:00 (in 9.999513 seconds)
notificationworker stdout | 2024-04-22 03:55:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:52 UTC)" (scheduled at 2024-04-22 03:55:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:55:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:55:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 42, 431901), True, datetime.datetime(2024, 4, 22, 3, 55, 42, 431901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:55:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:55:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:55:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:55:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:55:42,815 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:55:43,676 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:55:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:55:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:47.065745+00:00 (in 2.998165 seconds)
securityworker stdout | 2024-04-22 03:55:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:14 UTC)" (scheduled at 2024-04-22 03:55:44.067050+00:00)
securityworker stdout | 2024-04-22 03:55:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:55:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:55:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:55:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:55:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:55:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:55:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:55:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:55:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:55:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:55:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 50, 44, 77321), 1, 2])
securityworker stdout | 2024-04-22 03:55:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:55:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:55:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 50, 44, 77321), 1, 2])
securityworker stdout | 2024-04-22 03:55:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:55:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:55:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:55:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:55:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:55:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:55:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:55:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:55:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:55:45,632 [282] [DEBUG] [app] Starting request: urn:request:90c33241-fcb1-489d-9c21-3d322393c1d0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:45,632 [285] [DEBUG] [app] Starting request: urn:request:a33d348f-da37-4cf6-a917-39d2e1613f45 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:55:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:55:45,635 [283] [DEBUG] [app] Starting request: urn:request:db60d218-16d5-4188-bb4c-7227f08b89f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:45,636 [281] [DEBUG] [app] Starting request: urn:request:8eb3b8e8-8eb5-4ebf-b7e7-2817d1946b29 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:55:45,636 [283] [DEBUG] [app] Ending request: urn:request:db60d218-16d5-4188-bb4c-7227f08b89f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:db60d218-16d5-4188-bb4c-7227f08b89f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:55:45,636 [281] [DEBUG] [app] Ending request: urn:request:8eb3b8e8-8eb5-4ebf-b7e7-2817d1946b29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8eb3b8e8-8eb5-4ebf-b7e7-2817d1946b29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:55:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:55:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:55:45,639 [284] [DEBUG] [app] Starting request: urn:request:d90d4838-3284-40dd-b1ff-586fbd2bb0fa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:45,639 [284] [DEBUG] [app] Ending request: urn:request:d90d4838-3284-40dd-b1ff-586fbd2bb0fa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d90d4838-3284-40dd-b1ff-586fbd2bb0fa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:45,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:55:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:55:45,640 [286] [DEBUG] [app] Starting request: urn:request:522c74c2-8fe7-4545-84a4-388ceea963f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:55:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:45,640 [286] [DEBUG] [app] Ending request: urn:request:522c74c2-8fe7-4545-84a4-388ceea963f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:522c74c2-8fe7-4545-84a4-388ceea963f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:55:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:55:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:55:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:55:45,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:55:45,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:55:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:45,668 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:55:45,673 [282] [DEBUG] [app] Ending request: urn:request:90c33241-fcb1-489d-9c21-3d322393c1d0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90c33241-fcb1-489d-9c21-3d322393c1d0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:45,673 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:55:45,673 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.042 118 0.042)
gunicorn-web stdout | 2024-04-22 03:55:45,679 [285] [DEBUG] [app] Ending request: urn:request:a33d348f-da37-4cf6-a917-39d2e1613f45 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a33d348f-da37-4cf6-a917-39d2e1613f45', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:55:45,679 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:55:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.049 118 0.049)
gunicorn-web stdout | 2024-04-22 03:55:45,680 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:55:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:55:46,076 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:55:46,395 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:55:46,847 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:55:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:55:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:14.067050+00:00 (in 27.000751 seconds)
securityworker stdout | 2024-04-22 03:55:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:17 UTC)" (scheduled at 2024-04-22 03:55:47.065745+00:00)
securityworker stdout | 2024-04-22 03:55:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:55:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:55:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:55:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:55:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:55:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:55:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:55:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:55:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:10.248668+00:00 (in 23.002265 seconds)
namespacegcworker stdout | 2024-04-22 03:55:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:47 UTC)" (scheduled at 2024-04-22 03:55:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:55:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:55:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 47, 246840), True, datetime.datetime(2024, 4, 22, 3, 55, 47, 246840), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:55:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:55:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:55:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:55:47,601 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:55:48,502 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:55:49,444 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:55:49,677 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:55:49,680 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:55:49,683 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:55:49,967 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:55:51,109 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:55:51,150 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:55:51,229 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:55:51,231 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:55:51,241 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:55:51,246 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:55:51,248 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:55:51,254 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:55:51,260 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:55:51,264 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:55:51,362 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:55:51,365 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:55:51,372 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:55:51,374 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:55:51,380 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:55:51,428 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:55:51,644 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:55:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:55:54.433683+00:00 (in 2.002197 seconds)
notificationworker stdout | 2024-04-22 03:55:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:02 UTC)" (scheduled at 2024-04-22 03:55:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:55:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:55:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 52, 431900), True, datetime.datetime(2024, 4, 22, 3, 55, 52, 431900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:55:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:55:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:55:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:55:52,448 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 03:55:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:55:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:18.047111+00:00 (in 25.000992 seconds)
exportactionlogsworker stdout | 2024-04-22 03:55:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:53 UTC)" (scheduled at 2024-04-22 03:55:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:55:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:55:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 55, 53, 46365), True, datetime.datetime(2024, 4, 22, 3, 55, 53, 46365), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:55:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:55:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:55:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:55:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:55:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:02.430973+00:00 (in 7.996783 seconds)
notificationworker stdout | 2024-04-22 03:55:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:54 UTC)" (scheduled at 2024-04-22 03:55:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:55:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:55:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:56:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:55:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:55:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:28.130973+00:00 (in 29.999487 seconds)
buildlogsarchiver stdout | 2024-04-22 03:55:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:28 UTC)" (scheduled at 2024-04-22 03:55:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:55:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 55, 58, 131759), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:55:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:55:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:55:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:56:00,632 [286] [DEBUG] [app] Starting request: urn:request:a88b25c7-6c53-4ec0-9497-d8c0d55dbbf5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:00,632 [282] [DEBUG] [app] Starting request: urn:request:31fac66e-5ec0-481a-bc6d-a0a090576272 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:56:00,635 [275] [DEBUG] [app] Starting request: urn:request:87e12789-9d34-4c0c-9fd6-67ed8851900b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:00,636 [275] [DEBUG] [app] Ending request: urn:request:87e12789-9d34-4c0c-9fd6-67ed8851900b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:87e12789-9d34-4c0c-9fd6-67ed8851900b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:56:00,636 [283] [DEBUG] [app] Starting request: urn:request:61c25c6b-a7ed-4764-9c24-56d26fb6435a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:56:00,637 [283] [DEBUG] [app] Ending request: urn:request:61c25c6b-a7ed-4764-9c24-56d26fb6435a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:61c25c6b-a7ed-4764-9c24-56d26fb6435a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:56:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:56:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:00,639 [285] [DEBUG] [app] Starting request: urn:request:02eb382a-48c5-4787-a726-ea3e1bdf66ea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:00,639 [285] [DEBUG] [app] Ending request: urn:request:02eb382a-48c5-4787-a726-ea3e1bdf66ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:02eb382a-48c5-4787-a726-ea3e1bdf66ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:56:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:00,640 [284] [DEBUG] [app] Starting request: urn:request:4f6b2df2-65d5-4986-b978-803e087d9f3b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:00,641 [284] [DEBUG] [app] Ending request: urn:request:4f6b2df2-65d5-4986-b978-803e087d9f3b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4f6b2df2-65d5-4986-b978-803e087d9f3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:56:00,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:00,661 [286] [DEBUG] [app] Ending request: urn:request:a88b25c7-6c53-4ec0-9497-d8c0d55dbbf5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a88b25c7-6c53-4ec0-9497-d8c0d55dbbf5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:00,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:56:00,662 [282] [DEBUG] [app] Ending request: urn:request:31fac66e-5ec0-481a-bc6d-a0a090576272 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:31fac66e-5ec0-481a-bc6d-a0a090576272', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:00,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.030)
gcworker stdout | 2024-04-22 03:56:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:56:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:31.842666+00:00 (in 29.999460 seconds)
gcworker stdout | 2024-04-22 03:56:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:31 UTC)" (scheduled at 2024-04-22 03:56:01.842666+00:00)
gcworker stdout | 2024-04-22 03:56:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548561843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:56:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:56:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:56:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:56:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:56:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:09.033206+00:00 (in 6.997111 seconds)
repositorygcworker stdout | 2024-04-22 03:56:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:02 UTC)" (scheduled at 2024-04-22 03:56:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:56:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:56:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:56:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:12.430973+00:00 (in 9.999558 seconds)
notificationworker stdout | 2024-04-22 03:56:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:12 UTC)" (scheduled at 2024-04-22 03:56:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:56:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:56:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 2, 431797), True, datetime.datetime(2024, 4, 22, 3, 56, 2, 431797), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:56:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:56:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:56:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:56:04,620 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:56:05,901 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:56:08,485 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:56:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:56:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:02.035613+00:00 (in 53.001862 seconds)
repositorygcworker stdout | 2024-04-22 03:56:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:09 UTC)" (scheduled at 2024-04-22 03:56:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:56:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:56:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 9, 34203), True, datetime.datetime(2024, 4, 22, 3, 56, 9, 34203), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:56:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:56:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:56:09,050 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:56:09,570 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:56:09,937 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:56:10,060 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:56:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:56:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:40.238173+00:00 (in 29.999473 seconds)
autopruneworker stdout | 2024-04-22 03:56:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:40 UTC)" (scheduled at 2024-04-22 03:56:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:56:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758110247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:56:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:56:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:30.247852+00:00 (in 19.998739 seconds)
namespacegcworker stdout | 2024-04-22 03:56:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:10 UTC)" (scheduled at 2024-04-22 03:56:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:56:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:56:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:56:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:56:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:56:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:56:10,649 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:56:11,298 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:56:11,482 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:56:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:22.430973+00:00 (in 9.999521 seconds)
notificationworker stdout | 2024-04-22 03:56:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:22 UTC)" (scheduled at 2024-04-22 03:56:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:56:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:56:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 12, 431851), True, datetime.datetime(2024, 4, 22, 3, 56, 12, 431851), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:56:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:56:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:56:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:56:12,837 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:56:13,686 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:56:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:56:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:17.065745+00:00 (in 2.998124 seconds)
securityworker stdout | 2024-04-22 03:56:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:44 UTC)" (scheduled at 2024-04-22 03:56:14.067050+00:00)
securityworker stdout | 2024-04-22 03:56:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:56:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:56:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:56:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:56:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:56:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:56:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:56:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:56:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:56:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:56:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 51, 14, 77927), 1, 2])
securityworker stdout | 2024-04-22 03:56:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:56:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:56:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 51, 14, 77927), 1, 2])
securityworker stdout | 2024-04-22 03:56:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:56:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:56:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:56:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:56:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:56:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:56:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:56:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:56:15,632 [286] [DEBUG] [app] Starting request: urn:request:c2fae774-45a3-4553-a498-ebc55199d40e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:15,632 [282] [DEBUG] [app] Starting request: urn:request:c4e67c71-1830-425a-b5a7-2fb16fb2efcf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:56:15,635 [275] [DEBUG] [app] Starting request: urn:request:52494db8-09db-40ca-9daf-d0f7e4135a4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:15,635 [281] [DEBUG] [app] Starting request: urn:request:21d670e2-9b55-4615-8f22-9f11b7d62ed6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:15,636 [275] [DEBUG] [app] Ending request: urn:request:52494db8-09db-40ca-9daf-d0f7e4135a4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:52494db8-09db-40ca-9daf-d0f7e4135a4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:56:15,636 [281] [DEBUG] [app] Ending request: urn:request:21d670e2-9b55-4615-8f22-9f11b7d62ed6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:21d670e2-9b55-4615-8f22-9f11b7d62ed6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:56:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:56:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:15,639 [282] [DEBUG] [app] Starting request: urn:request:5755412f-29c3-4f28-8527-1ccae619fb71 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:15,639 [285] [DEBUG] [app] Starting request: urn:request:fe0f3777-c455-4848-a1d9-7822048151fe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:15,639 [282] [DEBUG] [app] Ending request: urn:request:5755412f-29c3-4f28-8527-1ccae619fb71 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5755412f-29c3-4f28-8527-1ccae619fb71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:56:15,639 [285] [DEBUG] [app] Ending request: urn:request:fe0f3777-c455-4848-a1d9-7822048151fe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe0f3777-c455-4848-a1d9-7822048151fe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:56:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:56:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:15,660 [282] [DEBUG] [app] Ending request: urn:request:c4e67c71-1830-425a-b5a7-2fb16fb2efcf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c4e67c71-1830-425a-b5a7-2fb16fb2efcf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.028)
gunicorn-web stdout | 2024-04-22 03:56:15,660 [286] [DEBUG] [app] Ending request: urn:request:c2fae774-45a3-4553-a498-ebc55199d40e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2fae774-45a3-4553-a498-ebc55199d40e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:15,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 03:56:16,113 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:56:16,431 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:56:16,879 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:56:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:56:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:44.067050+00:00 (in 27.000768 seconds)
securityworker stdout | 2024-04-22 03:56:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:47 UTC)" (scheduled at 2024-04-22 03:56:17.065745+00:00)
securityworker stdout | 2024-04-22 03:56:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:56:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:56:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:56:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:56:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:56:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:56:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:56:17,612 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:56:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:56:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:53.045554+00:00 (in 34.997877 seconds)
exportactionlogsworker stdout | 2024-04-22 03:56:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:18 UTC)" (scheduled at 2024-04-22 03:56:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:56:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:56:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:56:18,510 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:56:19,483 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:56:19,684 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:56:19,686 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:56:19,690 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:56:19,980 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
blobuploadcleanupworker stdout | 2024-04-22 03:56:21,139 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:56:21,158 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:56:21,239 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:56:21,241 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:56:21,248 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:56:21,258 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:56:21,261 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:56:21,264 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:56:21,270 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:56:21,273 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:56:21,369 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:56:21,372 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:56:21,379 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:56:21,381 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:56:21,386 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:56:21,463 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:56:21,680 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:56:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:32.430973+00:00 (in 9.999560 seconds)
notificationworker stdout | 2024-04-22 03:56:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:32 UTC)" (scheduled at 2024-04-22 03:56:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:56:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:56:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 22, 431853), True, datetime.datetime(2024, 4, 22, 3, 56, 22, 431853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:56:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:56:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:56:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:56:22,484 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 03:56:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:56:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:24.738365+00:00 (in 59.999501 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:56:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:24 UTC)" (scheduled at 2024-04-22 03:56:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:56:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:56:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:56:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:56:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:56:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:58.130973+00:00 (in 29.999467 seconds)
buildlogsarchiver stdout | 2024-04-22 03:56:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:58 UTC)" (scheduled at 2024-04-22 03:56:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:56:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 56, 28, 131755), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:56:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:56:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:56:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:56:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:31.933721+00:00 (in 2.002252 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:29 UTC)" (scheduled at 2024-04-22 03:56:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 29, 931735), True, datetime.datetime(2024, 4, 22, 3, 56, 29, 931735), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:56:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:56:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:56:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:47.245913+00:00 (in 16.997561 seconds)
namespacegcworker stdout | 2024-04-22 03:56:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:01:30 UTC)" (scheduled at 2024-04-22 03:56:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 03:56:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 56, 30, 248567), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:56:30,262 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 56, 30, 248567), True, datetime.datetime(2024, 4, 22, 3, 56, 30, 248567), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:56:30,267 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 56, 30, 248567), True, datetime.datetime(2024, 4, 22, 3, 56, 30, 248567), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 3, 56, 30, 248567), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 03:56:30,271 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:56:30,271 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:01:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:56:30,632 [284] [DEBUG] [app] Starting request: urn:request:efddaf84-8f2c-43e0-a32b-949314a65972 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:30,632 [282] [DEBUG] [app] Starting request: urn:request:b5bf125c-3343-49dd-9f93-97cc2cd08120 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:56:30,636 [279] [DEBUG] [app] Starting request: urn:request:dc3c793f-d130-4744-8a8c-84be769a8c78 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:30,636 [275] [DEBUG] [app] Starting request: urn:request:731d393b-2c7e-4f8d-bab6-8a25f037684a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:30,636 [279] [DEBUG] [app] Ending request: urn:request:dc3c793f-d130-4744-8a8c-84be769a8c78 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:dc3c793f-d130-4744-8a8c-84be769a8c78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:56:30,636 [275] [DEBUG] [app] Ending request: urn:request:731d393b-2c7e-4f8d-bab6-8a25f037684a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:731d393b-2c7e-4f8d-bab6-8a25f037684a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:56:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:56:30,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:56:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:30,639 [284] [DEBUG] [app] Starting request: urn:request:ef593d6d-8d51-4e1d-8473-de50d9bc7c00 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:30,639 [284] [DEBUG] [app] Ending request: urn:request:ef593d6d-8d51-4e1d-8473-de50d9bc7c00 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ef593d6d-8d51-4e1d-8473-de50d9bc7c00', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:56:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:30,640 [286] [DEBUG] [app] Starting request: urn:request:f2e8c761-e026-4e38-9ee6-9fdced9c4cbd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:56:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:30,640 [286] [DEBUG] [app] Ending request: urn:request:f2e8c761-e026-4e38-9ee6-9fdced9c4cbd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2e8c761-e026-4e38-9ee6-9fdced9c4cbd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:56:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:56:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:30,661 [284] [DEBUG] [app] Ending request: urn:request:efddaf84-8f2c-43e0-a32b-949314a65972 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:efddaf84-8f2c-43e0-a32b-949314a65972', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:30,661 [282] [DEBUG] [app] Ending request: urn:request:b5bf125c-3343-49dd-9f93-97cc2cd08120 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5bf125c-3343-49dd-9f93-97cc2cd08120', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:30,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:30,662 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 03:56:30,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.030)
gcworker stdout | 2024-04-22 03:56:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:56:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:01.842666+00:00 (in 29.999473 seconds)
gcworker stdout | 2024-04-22 03:56:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:01 UTC)" (scheduled at 2024-04-22 03:56:31.842666+00:00)
gcworker stdout | 2024-04-22 03:56:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548591843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:56:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:56:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:56:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:56:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:56:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:29.931002+00:00 (in 57.996769 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:56:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:31 UTC)" (scheduled at 2024-04-22 03:56:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:56:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:56:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:56:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:42.430973+00:00 (in 9.999546 seconds)
notificationworker stdout | 2024-04-22 03:56:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:42 UTC)" (scheduled at 2024-04-22 03:56:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:56:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:56:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 32, 431791), True, datetime.datetime(2024, 4, 22, 3, 56, 32, 431791), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:56:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:56:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:56:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:56:34,657 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:56:35,918 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:56:38,520 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:56:39,586 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:56:39,967 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:56:40,094 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:56:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:56:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:10.238173+00:00 (in 29.999477 seconds)
autopruneworker stdout | 2024-04-22 03:56:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:10 UTC)" (scheduled at 2024-04-22 03:56:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:56:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758140247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:56:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:56:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:56:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:56:40,682 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:56:41,309 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:56:41,511 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:56:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:52.430973+00:00 (in 9.999534 seconds)
notificationworker stdout | 2024-04-22 03:56:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:52 UTC)" (scheduled at 2024-04-22 03:56:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:56:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:56:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 42, 431876), True, datetime.datetime(2024, 4, 22, 3, 56, 42, 431876), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:56:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:56:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:56:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:56:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:56:42,873 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:56:43,723 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:56:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:56:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:47.065745+00:00 (in 2.998148 seconds)
securityworker stdout | 2024-04-22 03:56:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:14 UTC)" (scheduled at 2024-04-22 03:56:44.067050+00:00)
securityworker stdout | 2024-04-22 03:56:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:56:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:56:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:56:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:56:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:56:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:56:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:56:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:56:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:56:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:56:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 51, 44, 77673), 1, 2])
securityworker stdout | 2024-04-22 03:56:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:56:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:56:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 51, 44, 77673), 1, 2])
securityworker stdout | 2024-04-22 03:56:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:56:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:56:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:56:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:56:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:56:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:56:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:56:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:56:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:56:45,632 [286] [DEBUG] [app] Starting request: urn:request:cf8de1be-f68c-49ba-b4aa-f14934e2c752 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:45,632 [282] [DEBUG] [app] Starting request: urn:request:e7bd765b-1c23-4795-8d2e-99d2a15f85ea (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:56:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:56:45,635 [283] [DEBUG] [app] Starting request: urn:request:83198859-a2f5-4c0c-8476-6508f500feec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:45,635 [275] [DEBUG] [app] Starting request: urn:request:e0996178-1234-4be4-8d1c-40e7712edc57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:56:45,636 [283] [DEBUG] [app] Ending request: urn:request:83198859-a2f5-4c0c-8476-6508f500feec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:83198859-a2f5-4c0c-8476-6508f500feec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:56:45,636 [275] [DEBUG] [app] Ending request: urn:request:e0996178-1234-4be4-8d1c-40e7712edc57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e0996178-1234-4be4-8d1c-40e7712edc57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:56:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:56:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:56:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:56:45,639 [285] [DEBUG] [app] Starting request: urn:request:7d829337-7d45-41de-a4fc-98ef58705bc6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:45,639 [284] [DEBUG] [app] Starting request: urn:request:bc90efbc-b21d-474a-a5dc-c640c1d15629 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:56:45,639 [285] [DEBUG] [app] Ending request: urn:request:7d829337-7d45-41de-a4fc-98ef58705bc6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7d829337-7d45-41de-a4fc-98ef58705bc6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:56:45,639 [284] [DEBUG] [app] Ending request: urn:request:bc90efbc-b21d-474a-a5dc-c640c1d15629 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bc90efbc-b21d-474a-a5dc-c640c1d15629', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:56:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:56:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:45,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:56:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:56:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:56:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:56:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:56:45,661 [286] [DEBUG] [app] Ending request: urn:request:cf8de1be-f68c-49ba-b4aa-f14934e2c752 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf8de1be-f68c-49ba-b4aa-f14934e2c752', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:45,661 [282] [DEBUG] [app] Ending request: urn:request:e7bd765b-1c23-4795-8d2e-99d2a15f85ea (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e7bd765b-1c23-4795-8d2e-99d2a15f85ea', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:56:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:56:45,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:56:45,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:56:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:56:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
expiredappspecifictokenworker stdout | 2024-04-22 03:56:46,149 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:56:46,466 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:56:46,915 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:56:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:56:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:14.067050+00:00 (in 27.000742 seconds)
securityworker stdout | 2024-04-22 03:56:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:17 UTC)" (scheduled at 2024-04-22 03:56:47.065745+00:00)
securityworker stdout | 2024-04-22 03:56:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:56:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:56:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:56:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:56:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:56:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:56:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:56:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:56:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:10.248668+00:00 (in 23.002233 seconds)
namespacegcworker stdout | 2024-04-22 03:56:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:47 UTC)" (scheduled at 2024-04-22 03:56:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:56:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:56:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 47, 246737), True, datetime.datetime(2024, 4, 22, 3, 56, 47, 246737), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:56:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:56:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:56:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:56:47,627 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:56:48,547 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:56:49,498 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:56:49,691 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:56:49,694 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:56:49,698 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:56:50,016 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:56:51,167 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
blobuploadcleanupworker stdout | 2024-04-22 03:56:51,174 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:56:51,246 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:56:51,249 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:56:51,257 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:56:51,267 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:56:51,271 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:56:51,274 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:56:51,276 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:56:51,281 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:56:51,379 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:56:51,381 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:56:51,386 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:56:51,388 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:56:51,392 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:56:51,479 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:56:51,717 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:56:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:56:54.433683+00:00 (in 2.002232 seconds)
notificationworker stdout | 2024-04-22 03:56:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:02 UTC)" (scheduled at 2024-04-22 03:56:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:56:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:56:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 52, 431908), True, datetime.datetime(2024, 4, 22, 3, 56, 52, 431908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:56:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:56:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:56:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:56:52,520 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 03:56:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:56:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:18.047111+00:00 (in 25.001054 seconds)
exportactionlogsworker stdout | 2024-04-22 03:56:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:53 UTC)" (scheduled at 2024-04-22 03:56:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:56:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:56:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 56, 53, 46318), True, datetime.datetime(2024, 4, 22, 3, 56, 53, 46318), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:56:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:56:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:56:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:56:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:56:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:02.430973+00:00 (in 7.996820 seconds)
notificationworker stdout | 2024-04-22 03:56:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:54 UTC)" (scheduled at 2024-04-22 03:56:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:56:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:56:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:57:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:56:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:56:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:28.130973+00:00 (in 29.999496 seconds)
buildlogsarchiver stdout | 2024-04-22 03:56:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:28 UTC)" (scheduled at 2024-04-22 03:56:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:56:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 56, 58, 131691), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:56:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:56:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:56:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:57:00,632 [285] [DEBUG] [app] Starting request: urn:request:b7229276-1c22-462b-bfdb-3c2a58e4b33d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:00,632 [286] [DEBUG] [app] Starting request: urn:request:6cee6ecd-42e0-463a-bf0d-12200e1cdc2c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:00,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:57:00,635 [275] [DEBUG] [app] Starting request: urn:request:382aeab0-1862-4690-b64f-aadfe28cea8a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:00,635 [283] [DEBUG] [app] Starting request: urn:request:6cd9ee96-22b2-42a4-a485-f212419bdf8f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:00,636 [275] [DEBUG] [app] Ending request: urn:request:382aeab0-1862-4690-b64f-aadfe28cea8a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:382aeab0-1862-4690-b64f-aadfe28cea8a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:00,636 [283] [DEBUG] [app] Ending request: urn:request:6cd9ee96-22b2-42a4-a485-f212419bdf8f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6cd9ee96-22b2-42a4-a485-f212419bdf8f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:57:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:57:00,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:00,638 [285] [DEBUG] [app] Starting request: urn:request:b670d581-6451-4471-a417-1c0ae60ecc30 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:00,639 [285] [DEBUG] [app] Ending request: urn:request:b670d581-6451-4471-a417-1c0ae60ecc30 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b670d581-6451-4471-a417-1c0ae60ecc30', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:57:00,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:00,639 [282] [DEBUG] [app] Starting request: urn:request:9e152fae-e4af-4015-aa90-1454b7e9786b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:57:00,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:00,639 [282] [DEBUG] [app] Ending request: urn:request:9e152fae-e4af-4015-aa90-1454b7e9786b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9e152fae-e4af-4015-aa90-1454b7e9786b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:57:00,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:57:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:57:00,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:00,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:00,659 [285] [DEBUG] [app] Ending request: urn:request:b7229276-1c22-462b-bfdb-3c2a58e4b33d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7229276-1c22-462b-bfdb-3c2a58e4b33d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:00,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:00,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:57:00,660 [286] [DEBUG] [app] Ending request: urn:request:6cee6ecd-42e0-463a-bf0d-12200e1cdc2c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6cee6ecd-42e0-463a-bf0d-12200e1cdc2c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:00,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:00,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gcworker stdout | 2024-04-22 03:57:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:57:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:31.842666+00:00 (in 29.999484 seconds)
gcworker stdout | 2024-04-22 03:57:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:31 UTC)" (scheduled at 2024-04-22 03:57:01.842666+00:00)
gcworker stdout | 2024-04-22 03:57:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548621843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:57:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:57:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:57:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:57:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:57:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:09.033206+00:00 (in 6.997042 seconds)
repositorygcworker stdout | 2024-04-22 03:57:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:02 UTC)" (scheduled at 2024-04-22 03:57:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:57:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:57:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:57:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:12.430973+00:00 (in 9.999542 seconds)
notificationworker stdout | 2024-04-22 03:57:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:12 UTC)" (scheduled at 2024-04-22 03:57:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:57:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:57:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 2, 431871), True, datetime.datetime(2024, 4, 22, 3, 57, 2, 431871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:57:02,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:57:02,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:02,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:57:04,667 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:57:05,954 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:57:08,557 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:57:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:57:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:02.035613+00:00 (in 53.001877 seconds)
repositorygcworker stdout | 2024-04-22 03:57:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:09 UTC)" (scheduled at 2024-04-22 03:57:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:57:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:57:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 9, 34192), True, datetime.datetime(2024, 4, 22, 3, 57, 9, 34192), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:57:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:57:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:57:09,050 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:57:09,619 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:57:10,005 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:57:10,102 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:57:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:57:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:40.238173+00:00 (in 29.999506 seconds)
autopruneworker stdout | 2024-04-22 03:57:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:40 UTC)" (scheduled at 2024-04-22 03:57:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:57:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758170247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:57:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:57:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:47.245913+00:00 (in 36.996743 seconds)
namespacegcworker stdout | 2024-04-22 03:57:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:10 UTC)" (scheduled at 2024-04-22 03:57:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:57:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:57:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:57:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:57:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:57:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:57:10,718 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:57:11,316 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:57:11,547 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:57:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:22.430973+00:00 (in 9.999447 seconds)
notificationworker stdout | 2024-04-22 03:57:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:22 UTC)" (scheduled at 2024-04-22 03:57:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:57:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:57:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 12, 431840), True, datetime.datetime(2024, 4, 22, 3, 57, 12, 431840), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:57:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:57:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:57:12,909 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:57:13,759 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:57:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:57:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:17.065745+00:00 (in 2.998098 seconds)
securityworker stdout | 2024-04-22 03:57:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:44 UTC)" (scheduled at 2024-04-22 03:57:14.067050+00:00)
securityworker stdout | 2024-04-22 03:57:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:57:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:57:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:57:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:57:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:57:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:57:14,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:57:14,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:57:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:57:14,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:57:14,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 52, 14, 78789), 1, 2])
securityworker stdout | 2024-04-22 03:57:14,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:57:14,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:57:14,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 52, 14, 78789), 1, 2])
securityworker stdout | 2024-04-22 03:57:14,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:57:14,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:57:14,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:14,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:57:14,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:57:14,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:57:14,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:57:14,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:14,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:57:14,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:57:15,632 [286] [DEBUG] [app] Starting request: urn:request:335b9eab-4acd-4228-bda2-570c6f59d568 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:15,632 [282] [DEBUG] [app] Starting request: urn:request:1639c5c2-f89d-4e34-9bf2-dd96c18a5e54 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:57:15,636 [280] [DEBUG] [app] Starting request: urn:request:632677d8-ab39-4894-91a8-a605a04c6b06 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:15,636 [283] [DEBUG] [app] Starting request: urn:request:4eb7d14a-266f-4e30-955f-4f47f5d27b80 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:15,636 [280] [DEBUG] [app] Ending request: urn:request:632677d8-ab39-4894-91a8-a605a04c6b06 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:632677d8-ab39-4894-91a8-a605a04c6b06', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:15,636 [283] [DEBUG] [app] Ending request: urn:request:4eb7d14a-266f-4e30-955f-4f47f5d27b80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4eb7d14a-266f-4e30-955f-4f47f5d27b80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:15,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:57:15,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:57:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:15,639 [284] [DEBUG] [app] Starting request: urn:request:dc5efc16-8314-4c2a-af8c-3f102d85abd7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:15,639 [286] [DEBUG] [app] Starting request: urn:request:141da29d-2a05-43a6-87ff-b7a316132b3e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:15,640 [284] [DEBUG] [app] Ending request: urn:request:dc5efc16-8314-4c2a-af8c-3f102d85abd7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc5efc16-8314-4c2a-af8c-3f102d85abd7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:57:15,640 [286] [DEBUG] [app] Ending request: urn:request:141da29d-2a05-43a6-87ff-b7a316132b3e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:141da29d-2a05-43a6-87ff-b7a316132b3e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:57:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:57:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:57:15,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:57:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:15,661 [282] [DEBUG] [app] Ending request: urn:request:1639c5c2-f89d-4e34-9bf2-dd96c18a5e54 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1639c5c2-f89d-4e34-9bf2-dd96c18a5e54', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:15,661 [286] [DEBUG] [app] Ending request: urn:request:335b9eab-4acd-4228-bda2-570c6f59d568 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:335b9eab-4acd-4228-bda2-570c6f59d568', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:15,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:57:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 03:57:16,185 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:57:16,493 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:57:16,951 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:57:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:57:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:44.067050+00:00 (in 27.000744 seconds)
securityworker stdout | 2024-04-22 03:57:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:47 UTC)" (scheduled at 2024-04-22 03:57:17.065745+00:00)
securityworker stdout | 2024-04-22 03:57:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:57:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:57:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:57:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:57:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:57:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:57:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:57:17,664 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:57:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:57:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:23.046900+00:00 (in 4.999281 seconds)
exportactionlogsworker stdout | 2024-04-22 03:57:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:18 UTC)" (scheduled at 2024-04-22 03:57:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:57:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:57:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:57:18,578 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:57:19,523 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:57:19,699 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:57:19,701 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:57:19,704 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:57:20,053 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:57:21,174 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
blobuploadcleanupworker stdout | 2024-04-22 03:57:21,211 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:57:21,255 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:57:21,257 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:57:21,264 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:57:21,275 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:57:21,281 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:57:21,284 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:57:21,287 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 03:57:21,289 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 03:57:21,387 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:57:21,389 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:57:21,393 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:57:21,395 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:57:21,398 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:57:21,504 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:57:21,754 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:57:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:32.430973+00:00 (in 9.999488 seconds)
notificationworker stdout | 2024-04-22 03:57:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:32 UTC)" (scheduled at 2024-04-22 03:57:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:57:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:57:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 22, 431741), True, datetime.datetime(2024, 4, 22, 3, 57, 22, 431741), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:57:22,461 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:57:22,461 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:22,461 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:57:22,532 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 03:57:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:57:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:53.045554+00:00 (in 29.998145 seconds)
exportactionlogsworker stdout | 2024-04-22 03:57:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:02:23 UTC)" (scheduled at 2024-04-22 03:57:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 03:57:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 57, 23, 47612), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:57:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 57, 23, 47612), True, datetime.datetime(2024, 4, 22, 3, 57, 23, 47612), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:57:23,066 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 57, 23, 47612), True, datetime.datetime(2024, 4, 22, 3, 57, 23, 47612), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 3, 57, 23, 47612), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 03:57:23,070 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:57:23,070 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:02:23 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 03:57:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:57:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:24.738365+00:00 (in 59.999478 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:57:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:24 UTC)" (scheduled at 2024-04-22 03:57:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:57:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:57:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:57:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:57:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:57:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:58.130973+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2024-04-22 03:57:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:58 UTC)" (scheduled at 2024-04-22 03:57:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:57:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 57, 28, 131721), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:57:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:57:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:57:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:57:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:31.933721+00:00 (in 2.002201 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:29 UTC)" (scheduled at 2024-04-22 03:57:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 29, 931911), True, datetime.datetime(2024, 4, 22, 3, 57, 29, 931911), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:57:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:57:30,632 [286] [DEBUG] [app] Starting request: urn:request:11df67aa-901a-4f52-96b3-7614a064b316 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:30,632 [282] [DEBUG] [app] Starting request: urn:request:2615fded-9407-42c6-ab68-21e0bf80fedc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:57:30,636 [283] [DEBUG] [app] Starting request: urn:request:e8608fa2-fa17-437b-9d7d-d87d1180fe01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:30,636 [275] [DEBUG] [app] Starting request: urn:request:475eb98e-360f-4a80-bd10-e2f3bdee4a56 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:30,636 [283] [DEBUG] [app] Ending request: urn:request:e8608fa2-fa17-437b-9d7d-d87d1180fe01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e8608fa2-fa17-437b-9d7d-d87d1180fe01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:30,636 [275] [DEBUG] [app] Ending request: urn:request:475eb98e-360f-4a80-bd10-e2f3bdee4a56 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:475eb98e-360f-4a80-bd10-e2f3bdee4a56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:57:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 03:57:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:30,639 [286] [DEBUG] [app] Starting request: urn:request:64744502-83d0-4fb4-a52d-71086d0511f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:30,639 [286] [DEBUG] [app] Ending request: urn:request:64744502-83d0-4fb4-a52d-71086d0511f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:64744502-83d0-4fb4-a52d-71086d0511f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:57:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:30,640 [285] [DEBUG] [app] Starting request: urn:request:55141dd3-abb2-4ea1-99a9-90d4e50c803b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:30,640 [285] [DEBUG] [app] Ending request: urn:request:55141dd3-abb2-4ea1-99a9-90d4e50c803b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:55141dd3-abb2-4ea1-99a9-90d4e50c803b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:57:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:57:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:30,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:57:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:30,661 [282] [DEBUG] [app] Ending request: urn:request:2615fded-9407-42c6-ab68-21e0bf80fedc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2615fded-9407-42c6-ab68-21e0bf80fedc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:30,661 [286] [DEBUG] [app] Ending request: urn:request:11df67aa-901a-4f52-96b3-7614a064b316 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:11df67aa-901a-4f52-96b3-7614a064b316', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:30,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:57:30,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 03:57:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:57:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:01.842666+00:00 (in 29.999432 seconds)
gcworker stdout | 2024-04-22 03:57:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:01 UTC)" (scheduled at 2024-04-22 03:57:31.842666+00:00)
gcworker stdout | 2024-04-22 03:57:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548651843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:57:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:57:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:57:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:57:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:57:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:29.931002+00:00 (in 57.996840 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:57:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:31 UTC)" (scheduled at 2024-04-22 03:57:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:57:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:57:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:57:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:38.433394+00:00 (in 6.001958 seconds)
notificationworker stdout | 2024-04-22 03:57:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:42 UTC)" (scheduled at 2024-04-22 03:57:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:57:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:57:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 32, 431711), True, datetime.datetime(2024, 4, 22, 3, 57, 32, 431711), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:57:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:57:32,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:32,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:57:34,701 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:57:35,979 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
notificationworker stdout | 2024-04-22 03:57:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:42.430973+00:00 (in 3.997137 seconds)
notificationworker stdout | 2024-04-22 03:57:38,433 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:02:38 UTC)" (scheduled at 2024-04-22 03:57:38.433394+00:00)
notificationworker stdout | 2024-04-22 03:57:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 57, 38, 434054), 'notification/%'])
notificationworker stdout | 2024-04-22 03:57:38,448 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 57, 38, 434054), True, datetime.datetime(2024, 4, 22, 3, 57, 38, 434054), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 03:57:38,452 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 57, 38, 434054), True, datetime.datetime(2024, 4, 22, 3, 57, 38, 434054), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 3, 57, 38, 434054), 'notification/%'])
notificationworker stdout | 2024-04-22 03:57:38,456 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:38,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:02:38 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 03:57:38,591 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:57:39,656 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:57:40,017 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:57:40,139 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:57:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:57:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:10.238173+00:00 (in 29.999480 seconds)
autopruneworker stdout | 2024-04-22 03:57:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:10 UTC)" (scheduled at 2024-04-22 03:57:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:57:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758200247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:57:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:57:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:57:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:57:40,749 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:57:41,341 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:57:41,582 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:57:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:52.430973+00:00 (in 9.999555 seconds)
notificationworker stdout | 2024-04-22 03:57:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:52 UTC)" (scheduled at 2024-04-22 03:57:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:57:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:57:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 42, 431666), True, datetime.datetime(2024, 4, 22, 3, 57, 42, 431666), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:57:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:57:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:57:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:57:42,917 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:57:43,796 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:57:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:57:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:47.065745+00:00 (in 2.998123 seconds)
securityworker stdout | 2024-04-22 03:57:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:14 UTC)" (scheduled at 2024-04-22 03:57:44.067050+00:00)
securityworker stdout | 2024-04-22 03:57:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:57:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:57:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:57:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:57:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:57:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:57:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:57:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:57:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:57:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:57:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 52, 44, 78183), 1, 2])
securityworker stdout | 2024-04-22 03:57:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:57:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:57:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 52, 44, 78183), 1, 2])
securityworker stdout | 2024-04-22 03:57:44,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:57:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:57:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:57:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:57:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:57:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:57:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:57:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:57:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:57:45,632 [285] [DEBUG] [app] Starting request: urn:request:e5791652-3c8a-4834-bd40-f823d5d91080 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:45,632 [286] [DEBUG] [app] Starting request: urn:request:ac23943a-f957-4a93-bc9c-7860174f6dbb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:57:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:57:45,635 [275] [DEBUG] [app] Starting request: urn:request:20213984-976d-4163-9eda-3b9d5508cf02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:45,635 [281] [DEBUG] [app] Starting request: urn:request:27d8818f-9ba9-48e8-8e05-d6efd139aeaf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:57:45,636 [275] [DEBUG] [app] Ending request: urn:request:20213984-976d-4163-9eda-3b9d5508cf02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20213984-976d-4163-9eda-3b9d5508cf02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:45,636 [281] [DEBUG] [app] Ending request: urn:request:27d8818f-9ba9-48e8-8e05-d6efd139aeaf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:27d8818f-9ba9-48e8-8e05-d6efd139aeaf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:57:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:57:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 03:57:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:57:45,639 [285] [DEBUG] [app] Starting request: urn:request:4e741d1c-8e9d-4503-b033-66bf91ba005c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:45,639 [285] [DEBUG] [app] Ending request: urn:request:4e741d1c-8e9d-4503-b033-66bf91ba005c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4e741d1c-8e9d-4503-b033-66bf91ba005c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:57:45,639 [284] [DEBUG] [app] Starting request: urn:request:c11f9412-565b-4d97-8cf6-6d5e422ddfe7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:57:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:45,640 [284] [DEBUG] [app] Ending request: urn:request:c11f9412-565b-4d97-8cf6-6d5e422ddfe7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c11f9412-565b-4d97-8cf6-6d5e422ddfe7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 03:57:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:57:45,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:57:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:57:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:57:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 03:57:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:57:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:57:45,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:45,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:57:45,661 [285] [DEBUG] [app] Ending request: urn:request:e5791652-3c8a-4834-bd40-f823d5d91080 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5791652-3c8a-4834-bd40-f823d5d91080', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:45,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:45,661 [286] [DEBUG] [app] Ending request: urn:request:ac23943a-f957-4a93-bc9c-7860174f6dbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac23943a-f957-4a93-bc9c-7860174f6dbb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:57:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:57:45,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:57:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:57:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:57:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 03:57:46,222 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:57:46,529 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:57:46,983 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:57:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:57:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:14.067050+00:00 (in 27.000802 seconds)
securityworker stdout | 2024-04-22 03:57:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:17 UTC)" (scheduled at 2024-04-22 03:57:47.065745+00:00)
securityworker stdout | 2024-04-22 03:57:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:57:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:57:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:57:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:57:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:57:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:57:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:57:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:57:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:10.248668+00:00 (in 23.002271 seconds)
namespacegcworker stdout | 2024-04-22 03:57:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:47 UTC)" (scheduled at 2024-04-22 03:57:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:57:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:57:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 47, 246689), True, datetime.datetime(2024, 4, 22, 3, 57, 47, 246689), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:57:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:57:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:57:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:57:47,700 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:57:48,595 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:57:49,561 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:57:49,708 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:57:49,711 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:57:49,715 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:57:50,089 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:57:51,186 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
blobuploadcleanupworker stdout | 2024-04-22 03:57:51,240 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:57:51,263 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:57:51,266 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:57:51,271 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:57:51,280 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:57:51,290 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:57:51,292 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:57:51,296 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 03:57:51,299 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 03:57:51,395 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:57:51,397 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:57:51,399 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:57:51,402 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:57:51,405 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:57:51,540 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:57:51,791 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:57:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:57:54.433683+00:00 (in 2.002171 seconds)
notificationworker stdout | 2024-04-22 03:57:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:02 UTC)" (scheduled at 2024-04-22 03:57:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:57:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:57:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 52, 431823), True, datetime.datetime(2024, 4, 22, 3, 57, 52, 431823), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:57:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:57:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:57:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:57:52,567 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 03:57:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:57:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:18.047111+00:00 (in 25.001087 seconds)
exportactionlogsworker stdout | 2024-04-22 03:57:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:53 UTC)" (scheduled at 2024-04-22 03:57:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:57:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:57:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 57, 53, 46282), True, datetime.datetime(2024, 4, 22, 3, 57, 53, 46282), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:57:53,066 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:57:53,066 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:57:53,066 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:57:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:57:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:02.430973+00:00 (in 7.996815 seconds)
notificationworker stdout | 2024-04-22 03:57:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:54 UTC)" (scheduled at 2024-04-22 03:57:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:57:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:57:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:58:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:57:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:57:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:28.130973+00:00 (in 29.999486 seconds)
buildlogsarchiver stdout | 2024-04-22 03:57:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:28 UTC)" (scheduled at 2024-04-22 03:57:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:57:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 57, 58, 131706), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:57:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:57:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:57:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:58:00,632 [282] [DEBUG] [app] Starting request: urn:request:7d0ef665-5d8c-41bc-b4e0-e5d5ab0d8ff6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:00,632 [284] [DEBUG] [app] Starting request: urn:request:4ccb4cc1-d521-4dbf-926a-0f5be0a24b5d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:00,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:58:00,635 [283] [DEBUG] [app] Starting request: urn:request:449b9af1-82ab-4615-b92d-2583fb4f38ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:00,635 [279] [DEBUG] [app] Starting request: urn:request:2faa5e27-b5a2-49e8-8766-55ee63036740 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:00,635 [283] [DEBUG] [app] Ending request: urn:request:449b9af1-82ab-4615-b92d-2583fb4f38ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:449b9af1-82ab-4615-b92d-2583fb4f38ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:58:00,635 [279] [DEBUG] [app] Ending request: urn:request:2faa5e27-b5a2-49e8-8766-55ee63036740 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2faa5e27-b5a2-49e8-8766-55ee63036740', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:58:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:58:00,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:58:00,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:00,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:00,638 [286] [DEBUG] [app] Starting request: urn:request:2bc5edd1-490a-4c59-9e7f-2ad3de44fd49 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:00,638 [285] [DEBUG] [app] Starting request: urn:request:2a9d411d-9892-4f62-b643-648cc39bf482 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:00,639 [286] [DEBUG] [app] Ending request: urn:request:2bc5edd1-490a-4c59-9e7f-2ad3de44fd49 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2bc5edd1-490a-4c59-9e7f-2ad3de44fd49', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:00,639 [285] [DEBUG] [app] Ending request: urn:request:2a9d411d-9892-4f62-b643-648cc39bf482 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2a9d411d-9892-4f62-b643-648cc39bf482', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:00,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:58:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:58:00,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:00,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:00,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:00,657 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:00,660 [284] [DEBUG] [app] Ending request: urn:request:4ccb4cc1-d521-4dbf-926a-0f5be0a24b5d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ccb4cc1-d521-4dbf-926a-0f5be0a24b5d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:00,661 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:58:00,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:58:00,669 [282] [DEBUG] [app] Ending request: urn:request:7d0ef665-5d8c-41bc-b4e0-e5d5ab0d8ff6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d0ef665-5d8c-41bc-b4e0-e5d5ab0d8ff6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:00,669 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:58:00,669 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.038 118 0.038)
gcworker stdout | 2024-04-22 03:58:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:58:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:31.842666+00:00 (in 29.999485 seconds)
gcworker stdout | 2024-04-22 03:58:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:31 UTC)" (scheduled at 2024-04-22 03:58:01.842666+00:00)
gcworker stdout | 2024-04-22 03:58:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548681843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:58:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:58:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:58:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:58:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:58:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:09.033206+00:00 (in 6.997021 seconds)
repositorygcworker stdout | 2024-04-22 03:58:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:02 UTC)" (scheduled at 2024-04-22 03:58:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:58:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:58:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:58:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:12.430973+00:00 (in 9.999533 seconds)
notificationworker stdout | 2024-04-22 03:58:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:12 UTC)" (scheduled at 2024-04-22 03:58:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:58:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:58:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 2, 431714), True, datetime.datetime(2024, 4, 22, 3, 58, 2, 431714), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:58:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:58:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:58:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:58:04,737 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:58:06,016 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:58:08,627 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:58:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:58:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:26.035364+00:00 (in 17.001676 seconds)
repositorygcworker stdout | 2024-04-22 03:58:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:09 UTC)" (scheduled at 2024-04-22 03:58:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:58:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:58:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 9, 34134), True, datetime.datetime(2024, 4, 22, 3, 58, 9, 34134), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:58:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:58:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:58:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:58:09,685 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:58:10,049 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:58:10,146 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:58:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:58:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:40.238173+00:00 (in 29.999512 seconds)
autopruneworker stdout | 2024-04-22 03:58:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:40 UTC)" (scheduled at 2024-04-22 03:58:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:58:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758230247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:58:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:58:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:47.245913+00:00 (in 36.996807 seconds)
namespacegcworker stdout | 2024-04-22 03:58:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:10 UTC)" (scheduled at 2024-04-22 03:58:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:58:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:58:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:58:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:58:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:58:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:58:10,765 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:58:11,362 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:58:11,604 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,142 [65] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,143 [65] [INFO] [apscheduler.executors.default] Running job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2024-04-22 03:58:12 UTC)" (scheduled at 2024-04-22 03:58:12.142129+00:00)
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,143 [65] [DEBUG] [__main__] Garbage collecting expired app specific tokens with window: 1 day, 0:00:00
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,143 [65] [DEBUG] [peewee] ('DELETE FROM "appspecificauthtoken" WHERE ("appspecificauthtoken"."expiration" < %s)', [datetime.datetime(2024, 4, 21, 3, 58, 12, 143638)])
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,144 [65] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:58:12.142129+00:00 (in 3599.997788 seconds)
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,156 [65] [DEBUG] [data.database] Disconnecting from database.
expiredappspecifictokenworker stdout | 2024-04-22 03:58:12,156 [65] [INFO] [apscheduler.executors.default] Job "ExpiredAppSpecificTokenWorker._gc_expired_tokens (trigger: interval[1:00:00], next run at: 2024-04-22 04:58:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:58:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:22.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 03:58:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:22 UTC)" (scheduled at 2024-04-22 03:58:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:58:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:58:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 12, 431814), True, datetime.datetime(2024, 4, 22, 3, 58, 12, 431814), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:58:12,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:58:12,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:58:12,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:58:12,953 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:58:13,832 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:58:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:58:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:17.065745+00:00 (in 2.998118 seconds)
securityworker stdout | 2024-04-22 03:58:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:44 UTC)" (scheduled at 2024-04-22 03:58:14.067050+00:00)
securityworker stdout | 2024-04-22 03:58:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:58:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:58:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:58:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:58:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:58:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:58:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:58:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:58:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:58:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:58:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 53, 14, 77825), 1, 2])
securityworker stdout | 2024-04-22 03:58:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:58:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:58:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 53, 14, 77825), 1, 2])
securityworker stdout | 2024-04-22 03:58:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:58:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:58:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:58:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:58:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:58:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:58:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:58:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:58:15,632 [282] [DEBUG] [app] Starting request: urn:request:61fd04ba-cbf5-4a49-905f-3d237511fca4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:15,632 [285] [DEBUG] [app] Starting request: urn:request:3547ddf2-6fb4-4002-8ac2-7fd5989563e8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:58:15,635 [275] [DEBUG] [app] Starting request: urn:request:6c398f67-143c-41e7-b790-3dfaef4b463f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:15,635 [278] [DEBUG] [app] Starting request: urn:request:8d650ca9-1d5b-4bb0-be72-6405b70dd572 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:15,636 [278] [DEBUG] [app] Ending request: urn:request:8d650ca9-1d5b-4bb0-be72-6405b70dd572 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8d650ca9-1d5b-4bb0-be72-6405b70dd572', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:58:15,636 [275] [DEBUG] [app] Ending request: urn:request:6c398f67-143c-41e7-b790-3dfaef4b463f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6c398f67-143c-41e7-b790-3dfaef4b463f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:58:15,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:58:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:58:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:58:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:15,638 [282] [DEBUG] [app] Starting request: urn:request:2ace0861-58bd-49ab-b03f-29661e545dc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:15,639 [286] [DEBUG] [app] Starting request: urn:request:c82ae037-cff7-4934-afec-ff0024a5726f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:15,639 [282] [DEBUG] [app] Ending request: urn:request:2ace0861-58bd-49ab-b03f-29661e545dc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2ace0861-58bd-49ab-b03f-29661e545dc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:15,639 [286] [DEBUG] [app] Ending request: urn:request:c82ae037-cff7-4934-afec-ff0024a5726f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c82ae037-cff7-4934-afec-ff0024a5726f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,651 [60] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,651 [60] [INFO] [apscheduler.executors.default] Running job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2024-04-22 03:58:15 UTC)" (scheduled at 2024-04-22 03:58:15.650842+00:00)
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,651 [60] [DEBUG] [util.locking] Acquiring global lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,651 [60] [DEBUG] [redis_lock.acquire] Getting 'lock:BLOB_CLEANUP' ...
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,652 [60] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:58:15.650842+00:00 (in 3599.998671 seconds)
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,654 [60] [INFO] [redis_lock.acquire] Got lock for 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,654 [60] [DEBUG] [util.locking] Acquired lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,654 [60] [DEBUG] [__main__] Performing blob upload cleanup
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,655 [60] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."created" <= %s) LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 20, 3, 58, 15, 654468), 1, 0])
gunicorn-web stdout | 2024-04-22 03:58:15,660 [285] [DEBUG] [app] Ending request: urn:request:3547ddf2-6fb4-4002-8ac2-7fd5989563e8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3547ddf2-6fb4-4002-8ac2-7fd5989563e8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:58:15,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,668 [60] [DEBUG] [__main__] No additional stale blob uploads found
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,668 [60] [DEBUG] [data.database] Disconnecting from database.
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,668 [60] [DEBUG] [__main__] Cleaning partial uploads not applicable to storage location "default"
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,668 [60] [DEBUG] [util.locking] Releasing lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,668 [60] [DEBUG] [redis_lock.release] Releasing 'lock:BLOB_CLEANUP'.
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,670 [60] [DEBUG] [util.locking] Released lock BLOB_CLEANUP
blobuploadcleanupworker stdout | 2024-04-22 03:58:15,670 [60] [INFO] [apscheduler.executors.default] Job "BlobUploadCleanupWorker._try_cleanup_uploads (trigger: interval[1:00:00], next run at: 2024-04-22 04:58:15 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:58:15,670 [282] [DEBUG] [app] Ending request: urn:request:61fd04ba-cbf5-4a49-905f-3d237511fca4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:61fd04ba-cbf5-4a49-905f-3d237511fca4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:15,670 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:58:15,671 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.040)
expiredappspecifictokenworker stdout | 2024-04-22 03:58:16,258 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:58:16,566 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:58:17,020 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:58:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:58:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:44.067050+00:00 (in 27.000758 seconds)
securityworker stdout | 2024-04-22 03:58:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:47 UTC)" (scheduled at 2024-04-22 03:58:17.065745+00:00)
securityworker stdout | 2024-04-22 03:58:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:58:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:58:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:58:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:58:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:58:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:58:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:58:17,736 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:58:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:58:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:53.045554+00:00 (in 34.997926 seconds)
exportactionlogsworker stdout | 2024-04-22 03:58:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:18 UTC)" (scheduled at 2024-04-22 03:58:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:58:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:58:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:58:18,602 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:58:19,598 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:58:19,716 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:58:19,718 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:58:19,723 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:58:20,109 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:58:21,197 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
blobuploadcleanupworker stdout | 2024-04-22 03:58:21,251 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:58:21,270 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:58:21,276 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:58:21,283 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:58:21,290 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:58:21,297 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:58:21,303 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:58:21,308 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 03:58:21,311 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 03:58:21,403 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:58:21,405 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:58:21,408 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:58:21,411 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:58:21,414 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:58:21,575 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:58:21,812 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:58:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:32.430973+00:00 (in 9.999469 seconds)
notificationworker stdout | 2024-04-22 03:58:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:32 UTC)" (scheduled at 2024-04-22 03:58:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:58:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:58:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 22, 431799), True, datetime.datetime(2024, 4, 22, 3, 58, 22, 431799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:58:22,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:58:22,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:58:22,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:58:22,602 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 03:58:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:58:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:24.738365+00:00 (in 59.999468 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:58:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:24 UTC)" (scheduled at 2024-04-22 03:58:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:58:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:58:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:58:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:58:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:58:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:02.035613+00:00 (in 35.999759 seconds)
repositorygcworker stdout | 2024-04-22 03:58:26,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:03:26 UTC)" (scheduled at 2024-04-22 03:58:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 03:58:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 58, 26, 36263), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:58:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 58, 26, 36263), True, datetime.datetime(2024, 4, 22, 3, 58, 26, 36263), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:58:26,055 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 58, 26, 36263), True, datetime.datetime(2024, 4, 22, 3, 58, 26, 36263), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 3, 58, 26, 36263), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 03:58:26,059 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:58:26,059 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:03:26 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:58:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:58:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:58.130973+00:00 (in 29.999510 seconds)
buildlogsarchiver stdout | 2024-04-22 03:58:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:58 UTC)" (scheduled at 2024-04-22 03:58:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:58:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 58, 28, 131687), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:58:28,150 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:58:28,150 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:58:28,150 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:58:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:30.933445+00:00 (in 1.001943 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:29 UTC)" (scheduled at 2024-04-22 03:58:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 29, 931950), True, datetime.datetime(2024, 4, 22, 3, 58, 29, 931950), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:58:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:58:30,632 [286] [DEBUG] [app] Starting request: urn:request:0776c6d2-d43b-479d-84c9-41cbd93048d6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:30,632 [285] [DEBUG] [app] Starting request: urn:request:f319e46d-f755-4d91-aab2-42dddae61d2f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:58:30,635 [281] [DEBUG] [app] Starting request: urn:request:9d9cdb01-0696-4ca8-bab1-b2f28fe5c72b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:30,635 [275] [DEBUG] [app] Starting request: urn:request:061768d1-6c39-4c61-a915-7c67499e35ad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:30,635 [281] [DEBUG] [app] Ending request: urn:request:9d9cdb01-0696-4ca8-bab1-b2f28fe5c72b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d9cdb01-0696-4ca8-bab1-b2f28fe5c72b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:58:30,635 [275] [DEBUG] [app] Ending request: urn:request:061768d1-6c39-4c61-a915-7c67499e35ad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:061768d1-6c39-4c61-a915-7c67499e35ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:58:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:58:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:58:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:30,638 [284] [DEBUG] [app] Starting request: urn:request:229f6c7f-ef74-47b5-a00e-58fc4c5f089e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:30,638 [282] [DEBUG] [app] Starting request: urn:request:b757ade3-a73f-4a03-bbc3-52f7e60ce642 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:30,639 [284] [DEBUG] [app] Ending request: urn:request:229f6c7f-ef74-47b5-a00e-58fc4c5f089e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:229f6c7f-ef74-47b5-a00e-58fc4c5f089e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:30,639 [282] [DEBUG] [app] Ending request: urn:request:b757ade3-a73f-4a03-bbc3-52f7e60ce642 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b757ade3-a73f-4a03-bbc3-52f7e60ce642', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:30,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:30,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:30,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:30,659 [285] [DEBUG] [app] Ending request: urn:request:f319e46d-f755-4d91-aab2-42dddae61d2f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f319e46d-f755-4d91-aab2-42dddae61d2f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:30,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:58:30,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:58:30,660 [286] [DEBUG] [app] Ending request: urn:request:0776c6d2-d43b-479d-84c9-41cbd93048d6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0776c6d2-d43b-479d-84c9-41cbd93048d6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:58:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:58:30,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:31.933721+00:00 (in 0.999754 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:03:30 UTC)" (scheduled at 2024-04-22 03:58:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 3, 58, 30, 934212), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,947 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 58, 30, 934212), True, datetime.datetime(2024, 4, 22, 3, 58, 30, 934212), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,952 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 3, 58, 30, 934212), True, datetime.datetime(2024, 4, 22, 3, 58, 30, 934212), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 3, 58, 30, 934212), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,955 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:58:30,956 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:03:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:58:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:58:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:01.842666+00:00 (in 29.999414 seconds)
gcworker stdout | 2024-04-22 03:58:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:01 UTC)" (scheduled at 2024-04-22 03:58:31.842666+00:00)
gcworker stdout | 2024-04-22 03:58:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548711843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:58:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:58:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:58:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:58:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:58:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:29.931002+00:00 (in 57.996798 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:58:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:31 UTC)" (scheduled at 2024-04-22 03:58:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:58:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:58:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:58:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:42.430973+00:00 (in 9.999550 seconds)
notificationworker stdout | 2024-04-22 03:58:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:42 UTC)" (scheduled at 2024-04-22 03:58:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:58:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:58:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 32, 431659), True, datetime.datetime(2024, 4, 22, 3, 58, 32, 431659), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:58:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:58:32,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:58:32,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:58:34,766 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:58:36,023 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:58:38,662 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:58:39,701 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:58:40,085 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:58:40,152 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:58:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:58:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:10.238173+00:00 (in 29.999504 seconds)
autopruneworker stdout | 2024-04-22 03:58:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:10 UTC)" (scheduled at 2024-04-22 03:58:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:58:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758260247, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:58:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:58:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:58:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:58:40,794 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:58:41,399 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:58:41,634 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:58:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:52.430973+00:00 (in 9.999532 seconds)
notificationworker stdout | 2024-04-22 03:58:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:52 UTC)" (scheduled at 2024-04-22 03:58:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:58:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:58:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 42, 431703), True, datetime.datetime(2024, 4, 22, 3, 58, 42, 431703), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:58:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:58:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:58:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:58:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:58:42,979 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:58:43,838 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:58:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:58:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:47.065745+00:00 (in 2.998143 seconds)
securityworker stdout | 2024-04-22 03:58:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:14 UTC)" (scheduled at 2024-04-22 03:58:44.067050+00:00)
securityworker stdout | 2024-04-22 03:58:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:58:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:58:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:58:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:58:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:58:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:58:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:58:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:58:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:58:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:58:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 53, 44, 77520), 1, 2])
securityworker stdout | 2024-04-22 03:58:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:58:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:58:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 53, 44, 77520), 1, 2])
securityworker stdout | 2024-04-22 03:58:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:58:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:58:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:58:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:58:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:58:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:58:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:58:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:58:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:58:45,631 [286] [DEBUG] [app] Starting request: urn:request:a2fe0619-c026-4cda-8be3-a9d25159bc0e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:45,631 [282] [DEBUG] [app] Starting request: urn:request:14a71c8f-20c6-4891-b26b-a5a500e1dad6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:58:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:58:45,635 [283] [DEBUG] [app] Starting request: urn:request:d2003957-5329-405f-bc37-e35e420128b2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:45,635 [277] [DEBUG] [app] Starting request: urn:request:4f1dabd1-246a-4962-ab14-89a85e9b1aa0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:58:45,635 [283] [DEBUG] [app] Ending request: urn:request:d2003957-5329-405f-bc37-e35e420128b2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d2003957-5329-405f-bc37-e35e420128b2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:58:45,635 [277] [DEBUG] [app] Ending request: urn:request:4f1dabd1-246a-4962-ab14-89a85e9b1aa0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4f1dabd1-246a-4962-ab14-89a85e9b1aa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 03:58:45,635 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:58:45,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:58:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:58:45,638 [282] [DEBUG] [app] Starting request: urn:request:5d08c106-3a6b-4f1b-a342-7fba2e2997b6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:58:45,638 [282] [DEBUG] [app] Ending request: urn:request:5d08c106-3a6b-4f1b-a342-7fba2e2997b6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5d08c106-3a6b-4f1b-a342-7fba2e2997b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:45,638 [285] [DEBUG] [app] Starting request: urn:request:e9e36dd9-2de1-40c2-98d5-2e76feaef6ae (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.000 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:58:45,639 [285] [DEBUG] [app] Ending request: urn:request:e9e36dd9-2de1-40c2-98d5-2e76feaef6ae (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e9e36dd9-2de1-40c2-98d5-2e76feaef6ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:58:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:58:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:58:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:58:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:58:45,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:45,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:58:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:45,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:58:45,659 [286] [DEBUG] [app] Ending request: urn:request:a2fe0619-c026-4cda-8be3-a9d25159bc0e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2fe0619-c026-4cda-8be3-a9d25159bc0e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:58:45,660 [282] [DEBUG] [app] Ending request: urn:request:14a71c8f-20c6-4891-b26b-a5a500e1dad6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:14a71c8f-20c6-4891-b26b-a5a500e1dad6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:58:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:58:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 03:58:45,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:58:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:58:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 03:58:46,294 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:58:46,603 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 03:58:47,053 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:58:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:58:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:14.067050+00:00 (in 27.000796 seconds)
securityworker stdout | 2024-04-22 03:58:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:17 UTC)" (scheduled at 2024-04-22 03:58:47.065745+00:00)
securityworker stdout | 2024-04-22 03:58:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:58:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:58:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:58:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:58:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:58:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:58:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 03:58:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:58:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:10.248668+00:00 (in 23.002228 seconds)
namespacegcworker stdout | 2024-04-22 03:58:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:47 UTC)" (scheduled at 2024-04-22 03:58:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:58:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:58:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 47, 246714), True, datetime.datetime(2024, 4, 22, 3, 58, 47, 246714), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:58:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:58:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:58:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:58:47,759 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:58:48,638 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:58:49,610 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:58:49,723 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:58:49,727 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:58:49,729 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:58:50,119 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:58:51,208 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:58:51,277 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
blobuploadcleanupworker stdout | 2024-04-22 03:58:51,287 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:58:51,290 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:58:51,294 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:58:51,299 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:58:51,303 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:58:51,314 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:58:51,317 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 03:58:51,321 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 03:58:51,409 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:58:51,412 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:58:51,416 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:58:51,419 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:58:51,421 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:58:51,611 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:58:51,848 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:58:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:58:54.433683+00:00 (in 2.002209 seconds)
notificationworker stdout | 2024-04-22 03:58:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:02 UTC)" (scheduled at 2024-04-22 03:58:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:58:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:58:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 52, 431892), True, datetime.datetime(2024, 4, 22, 3, 58, 52, 431892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:58:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:58:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:58:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:58:52,639 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 03:58:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:58:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:18.047111+00:00 (in 25.001079 seconds)
exportactionlogsworker stdout | 2024-04-22 03:58:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:53 UTC)" (scheduled at 2024-04-22 03:58:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:58:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:58:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 58, 53, 46350), True, datetime.datetime(2024, 4, 22, 3, 58, 53, 46350), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:58:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:58:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:58:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:58:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:58:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:02.430973+00:00 (in 7.996802 seconds)
notificationworker stdout | 2024-04-22 03:58:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:54 UTC)" (scheduled at 2024-04-22 03:58:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:58:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:58:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 03:59:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:58:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:58:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:28.130973+00:00 (in 29.999529 seconds)
buildlogsarchiver stdout | 2024-04-22 03:58:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:28 UTC)" (scheduled at 2024-04-22 03:58:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:58:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 58, 58, 131660), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:58:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:58:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:58:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:59:00,632 [282] [DEBUG] [app] Starting request: urn:request:af8de863-27bf-4f0b-a44d-6870ac71c444 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:00,632 [286] [DEBUG] [app] Starting request: urn:request:51e218d1-a117-4b51-bbb1-31450d4666f2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:59:00,636 [281] [DEBUG] [app] Starting request: urn:request:03458d24-e91c-4053-8060-7b1408d94a9a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:00,636 [281] [DEBUG] [app] Ending request: urn:request:03458d24-e91c-4053-8060-7b1408d94a9a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03458d24-e91c-4053-8060-7b1408d94a9a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:59:00,636 [276] [DEBUG] [app] Starting request: urn:request:6286d995-7055-4238-bdbe-169b7149e189 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:00,636 [276] [DEBUG] [app] Ending request: urn:request:6286d995-7055-4238-bdbe-169b7149e189 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6286d995-7055-4238-bdbe-169b7149e189', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:59:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:59:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 03:59:00,637 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 03:59:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:00,639 [282] [DEBUG] [app] Starting request: urn:request:fe8f55b7-72a0-43ed-8680-131cdfe17edb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:00,639 [282] [DEBUG] [app] Ending request: urn:request:fe8f55b7-72a0-43ed-8680-131cdfe17edb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fe8f55b7-72a0-43ed-8680-131cdfe17edb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:59:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:00,640 [285] [DEBUG] [app] Starting request: urn:request:3d2831f4-286b-457b-910b-8ac7b796d2aa (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:00,640 [285] [DEBUG] [app] Ending request: urn:request:3d2831f4-286b-457b-910b-8ac7b796d2aa (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d2831f4-286b-457b-910b-8ac7b796d2aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:59:00,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:00,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:00,661 [282] [DEBUG] [app] Ending request: urn:request:af8de863-27bf-4f0b-a44d-6870ac71c444 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:af8de863-27bf-4f0b-a44d-6870ac71c444', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:59:00,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:59:00,662 [286] [DEBUG] [app] Ending request: urn:request:51e218d1-a117-4b51-bbb1-31450d4666f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:51e218d1-a117-4b51-bbb1-31450d4666f2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:59:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 03:59:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 03:59:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:59:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:31.842666+00:00 (in 29.999416 seconds)
gcworker stdout | 2024-04-22 03:59:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:31 UTC)" (scheduled at 2024-04-22 03:59:01.842666+00:00)
gcworker stdout | 2024-04-22 03:59:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548741843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:59:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:59:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:59:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 03:59:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:59:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:09.033206+00:00 (in 6.997053 seconds)
repositorygcworker stdout | 2024-04-22 03:59:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:02 UTC)" (scheduled at 2024-04-22 03:59:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 03:59:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 03:59:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:59:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:12.430973+00:00 (in 9.999533 seconds)
notificationworker stdout | 2024-04-22 03:59:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:12 UTC)" (scheduled at 2024-04-22 03:59:02.430973+00:00)
notificationworker stdout | 2024-04-22 03:59:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:59:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 2, 431853), True, datetime.datetime(2024, 4, 22, 3, 59, 2, 431853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:59:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:59:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:59:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:59:04,791 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:59:06,059 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:59:08,699 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 03:59:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 03:59:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:02.035613+00:00 (in 53.001868 seconds)
repositorygcworker stdout | 2024-04-22 03:59:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:09 UTC)" (scheduled at 2024-04-22 03:59:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 03:59:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 03:59:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 9, 34196), True, datetime.datetime(2024, 4, 22, 3, 59, 9, 34196), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 03:59:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 03:59:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 03:59:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:59:09,737 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:59:10,098 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:59:10,166 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:59:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:59:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:40.238173+00:00 (in 29.999559 seconds)
autopruneworker stdout | 2024-04-22 03:59:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:40 UTC)" (scheduled at 2024-04-22 03:59:10.238173+00:00)
autopruneworker stdout | 2024-04-22 03:59:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758290246, None, 1, 0])
namespacegcworker stdout | 2024-04-22 03:59:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:59:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:47.245913+00:00 (in 36.996752 seconds)
namespacegcworker stdout | 2024-04-22 03:59:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:10 UTC)" (scheduled at 2024-04-22 03:59:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 03:59:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 03:59:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 03:59:10,252 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:59:10,252 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:59:10,252 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:59:10,803 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:59:11,435 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:59:11,650 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:59:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:22.430973+00:00 (in 9.999499 seconds)
notificationworker stdout | 2024-04-22 03:59:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:22 UTC)" (scheduled at 2024-04-22 03:59:12.430973+00:00)
notificationworker stdout | 2024-04-22 03:59:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:59:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 12, 431862), True, datetime.datetime(2024, 4, 22, 3, 59, 12, 431862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:59:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:59:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:59:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:59:12,989 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:59:13,874 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:59:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:59:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:17.065745+00:00 (in 2.998117 seconds)
securityworker stdout | 2024-04-22 03:59:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:44 UTC)" (scheduled at 2024-04-22 03:59:14.067050+00:00)
securityworker stdout | 2024-04-22 03:59:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:59:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:59:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:59:14,070 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:59:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:59:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:59:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:59:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:59:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:59:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:59:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 54, 14, 76362), 1, 2])
securityworker stdout | 2024-04-22 03:59:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:59:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:59:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 54, 14, 76362), 1, 2])
securityworker stdout | 2024-04-22 03:59:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:59:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:59:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:59:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:59:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:59:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:59:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:59:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:59:15,632 [282] [DEBUG] [app] Starting request: urn:request:36be033b-e8d9-46c7-a23a-7cff695d1e82 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:15,632 [285] [DEBUG] [app] Starting request: urn:request:3aa47cd9-b343-4c29-bcc0-b81c7223cb62 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:59:15,635 [283] [DEBUG] [app] Starting request: urn:request:e5eb6fa3-5558-4f71-ad53-975b45546685 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:15,635 [275] [DEBUG] [app] Starting request: urn:request:d61484c5-9110-4f4f-a0b3-a16c83abd80f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:15,636 [275] [DEBUG] [app] Ending request: urn:request:d61484c5-9110-4f4f-a0b3-a16c83abd80f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d61484c5-9110-4f4f-a0b3-a16c83abd80f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:59:15,636 [283] [DEBUG] [app] Ending request: urn:request:e5eb6fa3-5558-4f71-ad53-975b45546685 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5eb6fa3-5558-4f71-ad53-975b45546685', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:59:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:59:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:15,638 [285] [DEBUG] [app] Starting request: urn:request:4efc2892-f23c-41fe-9dff-f95f7f5ec765 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:15,639 [286] [DEBUG] [app] Starting request: urn:request:246cc22d-b88d-4bee-8e02-df9212bf9132 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:15,639 [285] [DEBUG] [app] Ending request: urn:request:4efc2892-f23c-41fe-9dff-f95f7f5ec765 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4efc2892-f23c-41fe-9dff-f95f7f5ec765', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:15,639 [286] [DEBUG] [app] Ending request: urn:request:246cc22d-b88d-4bee-8e02-df9212bf9132 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:246cc22d-b88d-4bee-8e02-df9212bf9132', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:59:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:59:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:15,660 [285] [DEBUG] [app] Ending request: urn:request:3aa47cd9-b343-4c29-bcc0-b81c7223cb62 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3aa47cd9-b343-4c29-bcc0-b81c7223cb62', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:15,660 [282] [DEBUG] [app] Ending request: urn:request:36be033b-e8d9-46c7-a23a-7cff695d1e82 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:36be033b-e8d9-46c7-a23a-7cff695d1e82', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:59:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:59:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:59:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 03:59:16,331 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:59:16,640 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 03:59:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:59:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:44.067050+00:00 (in 27.000768 seconds)
securityworker stdout | 2024-04-22 03:59:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:47 UTC)" (scheduled at 2024-04-22 03:59:17.065745+00:00)
securityworker stdout | 2024-04-22 03:59:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:59:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
globalpromstats stdout | 2024-04-22 03:59:17,069 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
securityworker stdout | 2024-04-22 03:59:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:59:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:59:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:59:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:59:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:59:17,772 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 03:59:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:59:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:53.045554+00:00 (in 34.997923 seconds)
exportactionlogsworker stdout | 2024-04-22 03:59:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:18 UTC)" (scheduled at 2024-04-22 03:59:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 03:59:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 03:59:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 03:59:18,660 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:59:19,642 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:59:19,730 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:59:19,734 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:59:19,736 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:59:20,140 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:59:21,220 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:59:21,288 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:59:21,300 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:59:21,303 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:59:21,307 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:59:21,310 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
blobuploadcleanupworker stdout | 2024-04-22 03:59:21,324 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:59:21,326 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:59:21,329 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 03:59:21,331 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 03:59:21,418 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:59:21,421 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:59:21,423 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:59:21,426 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:59:21,429 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:59:21,646 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:59:21,874 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:59:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:32.430973+00:00 (in 9.999493 seconds)
notificationworker stdout | 2024-04-22 03:59:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:32 UTC)" (scheduled at 2024-04-22 03:59:22.430973+00:00)
notificationworker stdout | 2024-04-22 03:59:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:59:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 22, 431901), True, datetime.datetime(2024, 4, 22, 3, 59, 22, 431901), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:59:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:59:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:59:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:59:22,675 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 03:59:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 03:59:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:24.738365+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2024-04-22 03:59:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:24 UTC)" (scheduled at 2024-04-22 03:59:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 03:59:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 03:59:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 03:59:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:59:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:59:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:58.130973+00:00 (in 29.999482 seconds)
buildlogsarchiver stdout | 2024-04-22 03:59:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:58 UTC)" (scheduled at 2024-04-22 03:59:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:59:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 59, 28, 131879), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:59:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:59:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:59:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 03:59:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:31.933721+00:00 (in 2.002207 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:29 UTC)" (scheduled at 2024-04-22 03:59:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 29, 931831), True, datetime.datetime(2024, 4, 22, 3, 59, 29, 931831), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 03:59:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:59:30,632 [282] [DEBUG] [app] Starting request: urn:request:28db32b4-c277-4643-b7a5-50fd4fab57f3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:30,632 [284] [DEBUG] [app] Starting request: urn:request:8a907a7c-9fd2-421d-ae39-659f443fcf94 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:30,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:59:30,635 [279] [DEBUG] [app] Starting request: urn:request:3b70e1bb-18c8-4114-b064-771da1417293 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:30,635 [283] [DEBUG] [app] Starting request: urn:request:9cb58aae-5a92-4efe-8f56-781fdef33803 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:30,636 [279] [DEBUG] [app] Ending request: urn:request:3b70e1bb-18c8-4114-b064-771da1417293 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b70e1bb-18c8-4114-b064-771da1417293', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:59:30,636 [283] [DEBUG] [app] Ending request: urn:request:9cb58aae-5a92-4efe-8f56-781fdef33803 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9cb58aae-5a92-4efe-8f56-781fdef33803', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:59:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:59:30,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 03:59:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:30,639 [282] [DEBUG] [app] Starting request: urn:request:0eb21fb3-3902-4bc7-b923-de49575706da (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:30,639 [282] [DEBUG] [app] Ending request: urn:request:0eb21fb3-3902-4bc7-b923-de49575706da (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0eb21fb3-3902-4bc7-b923-de49575706da', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:30,639 [286] [DEBUG] [app] Starting request: urn:request:79b75edc-c30c-4333-b47d-1c902c296c2f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:59:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:30,639 [286] [DEBUG] [app] Ending request: urn:request:79b75edc-c30c-4333-b47d-1c902c296c2f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:79b75edc-c30c-4333-b47d-1c902c296c2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:59:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:30,660 [284] [DEBUG] [app] Ending request: urn:request:8a907a7c-9fd2-421d-ae39-659f443fcf94 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8a907a7c-9fd2-421d-ae39-659f443fcf94', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:30,661 [282] [DEBUG] [app] Ending request: urn:request:28db32b4-c277-4643-b7a5-50fd4fab57f3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28db32b4-c277-4643-b7a5-50fd4fab57f3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:59:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:59:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:59:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 03:59:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 03:59:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:01.842666+00:00 (in 29.999440 seconds)
gcworker stdout | 2024-04-22 03:59:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:01 UTC)" (scheduled at 2024-04-22 03:59:31.842666+00:00)
gcworker stdout | 2024-04-22 03:59:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548771843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 03:59:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 03:59:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 03:59:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:59:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 03:59:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:29.931002+00:00 (in 57.996778 seconds)
securityscanningnotificationworker stdout | 2024-04-22 03:59:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:31 UTC)" (scheduled at 2024-04-22 03:59:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 03:59:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 03:59:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:59:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:42.430973+00:00 (in 9.999489 seconds)
notificationworker stdout | 2024-04-22 03:59:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:42 UTC)" (scheduled at 2024-04-22 03:59:32.430973+00:00)
notificationworker stdout | 2024-04-22 03:59:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:59:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 32, 431926), True, datetime.datetime(2024, 4, 22, 3, 59, 32, 431926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:59:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:59:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:59:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:59:34,812 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 03:59:36,078 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 03:59:38,735 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 03:59:39,774 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 03:59:40,134 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 03:59:40,173 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 03:59:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 03:59:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:10.238173+00:00 (in 29.999493 seconds)
autopruneworker stdout | 2024-04-22 03:59:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:10 UTC)" (scheduled at 2024-04-22 03:59:40.238173+00:00)
autopruneworker stdout | 2024-04-22 03:59:40,250 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758320250, None, 1, 0])
autopruneworker stdout | 2024-04-22 03:59:40,256 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 03:59:40,256 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 03:59:40,256 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 03:59:40,837 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 03:59:41,448 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 03:59:41,660 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 03:59:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:52.430973+00:00 (in 9.999460 seconds)
notificationworker stdout | 2024-04-22 03:59:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:52 UTC)" (scheduled at 2024-04-22 03:59:42.430973+00:00)
notificationworker stdout | 2024-04-22 03:59:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:59:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 42, 431948), True, datetime.datetime(2024, 4, 22, 3, 59, 42, 431948), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:59:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:59:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:59:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 03:59:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 03:59:43,024 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 03:59:43,911 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 03:59:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:59:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:47.065745+00:00 (in 2.998145 seconds)
securityworker stdout | 2024-04-22 03:59:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:14 UTC)" (scheduled at 2024-04-22 03:59:44.067050+00:00)
securityworker stdout | 2024-04-22 03:59:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 03:59:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:59:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 03:59:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:59:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:59:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:59:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:59:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 03:59:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:59:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:59:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 54, 44, 77914), 1, 2])
securityworker stdout | 2024-04-22 03:59:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:59:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:59:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 54, 44, 77914), 1, 2])
securityworker stdout | 2024-04-22 03:59:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 03:59:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 03:59:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 03:59:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 03:59:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 03:59:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 03:59:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 03:59:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:59:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 03:59:45,632 [284] [DEBUG] [app] Starting request: urn:request:6774da3b-0740-4d53-860a-f69d2a8c153a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:45,632 [282] [DEBUG] [app] Starting request: urn:request:8640c164-a791-4ff8-a2ef-9e8f7c452050 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 03:59:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 03:59:45,636 [283] [DEBUG] [app] Starting request: urn:request:5923aa02-a7b7-455a-a174-fba5ba67046c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:45,636 [281] [DEBUG] [app] Starting request: urn:request:a6476a18-96aa-4468-a807-5e7286a43db7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 03:59:45,636 [283] [DEBUG] [app] Ending request: urn:request:5923aa02-a7b7-455a-a174-fba5ba67046c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5923aa02-a7b7-455a-a174-fba5ba67046c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 03:59:45,636 [281] [DEBUG] [app] Ending request: urn:request:a6476a18-96aa-4468-a807-5e7286a43db7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a6476a18-96aa-4468-a807-5e7286a43db7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 03:59:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 03:59:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 03:59:45,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 03:59:45,639 [282] [DEBUG] [app] Starting request: urn:request:815b376e-d7d6-4ddc-9d1b-fc69b578461d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:45,639 [282] [DEBUG] [app] Ending request: urn:request:815b376e-d7d6-4ddc-9d1b-fc69b578461d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:815b376e-d7d6-4ddc-9d1b-fc69b578461d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:45,639 [286] [DEBUG] [app] Starting request: urn:request:47ee2e49-906a-495d-8795-9de2e561ccfd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 03:59:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:45,639 [286] [DEBUG] [app] Ending request: urn:request:47ee2e49-906a-495d-8795-9de2e561ccfd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:47ee2e49-906a-495d-8795-9de2e561ccfd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 03:59:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 03:59:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:03:59:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 03:59:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:03:59:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 03:59:45,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 03:59:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 03:59:45,660 [284] [DEBUG] [app] Ending request: urn:request:6774da3b-0740-4d53-860a-f69d2a8c153a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6774da3b-0740-4d53-860a-f69d2a8c153a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:45,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 03:59:45,660 [282] [DEBUG] [app] Ending request: urn:request:8640c164-a791-4ff8-a2ef-9e8f7c452050 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8640c164-a791-4ff8-a2ef-9e8f7c452050', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 03:59:45,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 03:59:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 03:59:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:03:59:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:03:59:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 03:59:46,368 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 03:59:46,677 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 03:59:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 03:59:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:14.067050+00:00 (in 27.000740 seconds)
securityworker stdout | 2024-04-22 03:59:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:17 UTC)" (scheduled at 2024-04-22 03:59:47.065745+00:00)
securityworker stdout | 2024-04-22 03:59:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 03:59:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 03:59:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 03:59:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:59:47,090 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 03:59:47,093 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 03:59:47,093 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:17 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 03:59:47,096 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
namespacegcworker stdout | 2024-04-22 03:59:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 03:59:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:10.248668+00:00 (in 23.002248 seconds)
namespacegcworker stdout | 2024-04-22 03:59:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:47 UTC)" (scheduled at 2024-04-22 03:59:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 03:59:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 03:59:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 47, 246688), True, datetime.datetime(2024, 4, 22, 3, 59, 47, 246688), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 03:59:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 03:59:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 03:59:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 03:59:47,789 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 03:59:48,690 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 03:59:49,679 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 03:59:49,738 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 03:59:49,741 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 03:59:49,743 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 03:59:50,177 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 03:59:51,230 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 03:59:51,299 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 03:59:51,308 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 03:59:51,313 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 03:59:51,317 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 03:59:51,319 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 03:59:51,336 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 03:59:51,339 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
blobuploadcleanupworker stdout | 2024-04-22 03:59:51,341 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-registry stdout | 2024-04-22 03:59:51,345 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 03:59:51,425 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 03:59:51,428 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 03:59:51,431 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 03:59:51,435 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 03:59:51,437 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 03:59:51,680 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 03:59:51,910 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 03:59:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 03:59:54.433683+00:00 (in 2.002202 seconds)
notificationworker stdout | 2024-04-22 03:59:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:02 UTC)" (scheduled at 2024-04-22 03:59:52.430973+00:00)
notificationworker stdout | 2024-04-22 03:59:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 03:59:52,433 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 52, 431908), True, datetime.datetime(2024, 4, 22, 3, 59, 52, 431908), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 03:59:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 03:59:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 03:59:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 03:59:52,685 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 03:59:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 03:59:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:18.047111+00:00 (in 25.001068 seconds)
exportactionlogsworker stdout | 2024-04-22 03:59:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:53 UTC)" (scheduled at 2024-04-22 03:59:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 03:59:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 03:59:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 3, 59, 53, 46326), True, datetime.datetime(2024, 4, 22, 3, 59, 53, 46326), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 03:59:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 03:59:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 03:59:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 03:59:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 03:59:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:02.430973+00:00 (in 7.996772 seconds)
notificationworker stdout | 2024-04-22 03:59:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:54 UTC)" (scheduled at 2024-04-22 03:59:54.433683+00:00)
notificationworker stdout | 2024-04-22 03:59:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 03:59:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:00:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 03:59:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 03:59:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:28.130973+00:00 (in 29.999529 seconds)
buildlogsarchiver stdout | 2024-04-22 03:59:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:28 UTC)" (scheduled at 2024-04-22 03:59:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 03:59:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 3, 59, 58, 131668), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 03:59:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 03:59:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 03:59:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:00:00,632 [286] [DEBUG] [app] Starting request: urn:request:e1b29cf5-924b-4f19-a48f-5a963355cbbb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:00,632 [282] [DEBUG] [app] Starting request: urn:request:0b76d598-1072-493b-adfa-d558cea16e4e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:00:00,635 [275] [DEBUG] [app] Starting request: urn:request:2cdacf98-b19d-4a86-ac8e-23b77b94ca57 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:00,635 [283] [DEBUG] [app] Starting request: urn:request:c513fd14-13f4-4b0e-bcfe-e6a7d8939dfa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:00,636 [275] [DEBUG] [app] Ending request: urn:request:2cdacf98-b19d-4a86-ac8e-23b77b94ca57 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2cdacf98-b19d-4a86-ac8e-23b77b94ca57', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:00:00,636 [283] [DEBUG] [app] Ending request: urn:request:c513fd14-13f4-4b0e-bcfe-e6a7d8939dfa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c513fd14-13f4-4b0e-bcfe-e6a7d8939dfa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:00:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:00:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:00,638 [282] [DEBUG] [app] Starting request: urn:request:d999fd4e-201f-482a-b30a-4d7fdab23ed8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:00,639 [282] [DEBUG] [app] Ending request: urn:request:d999fd4e-201f-482a-b30a-4d7fdab23ed8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d999fd4e-201f-482a-b30a-4d7fdab23ed8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:00,639 [285] [DEBUG] [app] Starting request: urn:request:81868200-f21c-44f1-9820-ead2099ded54 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:00,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:00,639 [285] [DEBUG] [app] Ending request: urn:request:81868200-f21c-44f1-9820-ead2099ded54 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:81868200-f21c-44f1-9820-ead2099ded54', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:00:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:00,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:00:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:00,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:00,660 [282] [DEBUG] [app] Ending request: urn:request:0b76d598-1072-493b-adfa-d558cea16e4e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0b76d598-1072-493b-adfa-d558cea16e4e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:00:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:00:00,661 [286] [DEBUG] [app] Ending request: urn:request:e1b29cf5-924b-4f19-a48f-5a963355cbbb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1b29cf5-924b-4f19-a48f-5a963355cbbb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:00:00,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:00:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:00:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:31.842666+00:00 (in 29.999428 seconds)
gcworker stdout | 2024-04-22 04:00:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:31 UTC)" (scheduled at 2024-04-22 04:00:01.842666+00:00)
gcworker stdout | 2024-04-22 04:00:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548801843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:00:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:00:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:00:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:00:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:00:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:09.033206+00:00 (in 6.997025 seconds)
repositorygcworker stdout | 2024-04-22 04:00:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:02 UTC)" (scheduled at 2024-04-22 04:00:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:00:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:00:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:00:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:12.430973+00:00 (in 9.999547 seconds)
notificationworker stdout | 2024-04-22 04:00:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:12 UTC)" (scheduled at 2024-04-22 04:00:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:00:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:00:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 2, 431830), True, datetime.datetime(2024, 4, 22, 4, 0, 2, 431830), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:00:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:00:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:00:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:00:04,848 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:00:06,114 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:00:08,771 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 04:00:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:00:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:02.035613+00:00 (in 53.001884 seconds)
repositorygcworker stdout | 2024-04-22 04:00:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:09 UTC)" (scheduled at 2024-04-22 04:00:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:00:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:00:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 9, 34175), True, datetime.datetime(2024, 4, 22, 4, 0, 9, 34175), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:00:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:00:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:00:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:00:09,804 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:00:10,167 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:00:10,209 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 04:00:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:00:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:40.238173+00:00 (in 29.999470 seconds)
autopruneworker stdout | 2024-04-22 04:00:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:40 UTC)" (scheduled at 2024-04-22 04:00:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:00:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758350247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:00:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:00:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:47.245913+00:00 (in 36.996818 seconds)
namespacegcworker stdout | 2024-04-22 04:00:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:10 UTC)" (scheduled at 2024-04-22 04:00:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:00:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:00:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:00:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:00:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:00:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:40 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:00:10,862 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:00:11,479 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:00:11,697 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:00:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:22.430973+00:00 (in 9.999485 seconds)
notificationworker stdout | 2024-04-22 04:00:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:22 UTC)" (scheduled at 2024-04-22 04:00:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:00:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:00:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 12, 431923), True, datetime.datetime(2024, 4, 22, 4, 0, 12, 431923), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:00:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:00:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:00:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:00:13,060 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:00:13,949 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:00:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:00:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:17.065745+00:00 (in 2.998138 seconds)
securityworker stdout | 2024-04-22 04:00:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:44 UTC)" (scheduled at 2024-04-22 04:00:14.067050+00:00)
securityworker stdout | 2024-04-22 04:00:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:00:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:00:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:00:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:14,073 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:00:14,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:00:14,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:00:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:00:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:00:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:00:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:00:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 55, 14, 78278), 1, 2])
securityworker stdout | 2024-04-22 04:00:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:00:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:00:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 55, 14, 78278), 1, 2])
securityworker stdout | 2024-04-22 04:00:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:00:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:00:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:00:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:00:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:00:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:00:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:00:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:00:15,632 [286] [DEBUG] [app] Starting request: urn:request:5548dc79-4bf1-4b0b-a230-aad44f705eb2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:15,632 [284] [DEBUG] [app] Starting request: urn:request:167a5cc4-d7d2-4098-8655-b3902e262a13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:00:15,635 [275] [DEBUG] [app] Starting request: urn:request:34405d1d-0894-4bc2-b443-84ce836a0de4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:15,635 [275] [DEBUG] [app] Ending request: urn:request:34405d1d-0894-4bc2-b443-84ce836a0de4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34405d1d-0894-4bc2-b443-84ce836a0de4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:00:15,635 [283] [DEBUG] [app] Starting request: urn:request:38992a03-e479-489c-b8c0-1e16c9c9ab4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:00:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:00:15,636 [283] [DEBUG] [app] Ending request: urn:request:38992a03-e479-489c-b8c0-1e16c9c9ab4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:38992a03-e479-489c-b8c0-1e16c9c9ab4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 04:00:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:00:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:15,638 [282] [DEBUG] [app] Starting request: urn:request:8d510ad6-a5f8-4587-acd9-0a750851418b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:15,639 [282] [DEBUG] [app] Ending request: urn:request:8d510ad6-a5f8-4587-acd9-0a750851418b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8d510ad6-a5f8-4587-acd9-0a750851418b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:00:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:15,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:15,640 [282] [DEBUG] [app] Starting request: urn:request:ede9ac3b-d59f-47b4-b874-f1ad48017bee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:15,640 [282] [DEBUG] [app] Ending request: urn:request:ede9ac3b-d59f-47b4-b874-f1ad48017bee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ede9ac3b-d59f-47b4-b874-f1ad48017bee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:00:15,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:15,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:15,659 [284] [DEBUG] [app] Ending request: urn:request:167a5cc4-d7d2-4098-8655-b3902e262a13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:167a5cc4-d7d2-4098-8655-b3902e262a13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:00:15,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:00:15,660 [286] [DEBUG] [app] Ending request: urn:request:5548dc79-4bf1-4b0b-a230-aad44f705eb2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5548dc79-4bf1-4b0b-a230-aad44f705eb2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:15,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:00:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 04:00:16,404 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:00:16,685 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:00:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:00:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:44.067050+00:00 (in 27.000768 seconds)
securityworker stdout | 2024-04-22 04:00:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:47 UTC)" (scheduled at 2024-04-22 04:00:17.065745+00:00)
securityworker stdout | 2024-04-22 04:00:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:00:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:00:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:00:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:00:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:00:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:00:17,087 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:00:17,119 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:00:17,805 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 04:00:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:00:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:53.045554+00:00 (in 34.997930 seconds)
exportactionlogsworker stdout | 2024-04-22 04:00:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:18 UTC)" (scheduled at 2024-04-22 04:00:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:00:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:00:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:00:18,728 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
securityworker stdout | 2024-04-22 04:00:19,716 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 04:00:19,743 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:00:19,749 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:00:19,751 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 04:00:20,194 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:00:21,239 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:00:21,308 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:00:21,317 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:00:21,321 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:00:21,327 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:00:21,329 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:00:21,345 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:00:21,350 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:00:21,353 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
blobuploadcleanupworker stdout | 2024-04-22 04:00:21,376 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-web stdout | 2024-04-22 04:00:21,433 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:00:21,437 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:00:21,440 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:00:21,442 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:00:21,445 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 04:00:21,709 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:00:21,926 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:00:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:32.430973+00:00 (in 9.999439 seconds)
notificationworker stdout | 2024-04-22 04:00:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:32 UTC)" (scheduled at 2024-04-22 04:00:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:00:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:00:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 22, 431941), True, datetime.datetime(2024, 4, 22, 4, 0, 22, 431941), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:00:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:00:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:00:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:00:22,720 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:00:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:00:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:24.738365+00:00 (in 59.999500 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:00:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:24 UTC)" (scheduled at 2024-04-22 04:00:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:00:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:00:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:00:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:00:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:00:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:58.130973+00:00 (in 29.999503 seconds)
buildlogsarchiver stdout | 2024-04-22 04:00:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:58 UTC)" (scheduled at 2024-04-22 04:00:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:00:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 0, 28, 131673), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:00:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:00:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:00:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:00:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:31.933721+00:00 (in 2.002194 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:29 UTC)" (scheduled at 2024-04-22 04:00:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 29, 931831), True, datetime.datetime(2024, 4, 22, 4, 0, 29, 931831), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:00:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:00:30,632 [284] [DEBUG] [app] Starting request: urn:request:99907522-abbe-4da7-bd4a-547f595e602a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:30,632 [282] [DEBUG] [app] Starting request: urn:request:960be804-5d1d-4bea-9d07-38bb94836158 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:00:30,635 [275] [DEBUG] [app] Starting request: urn:request:30edac6b-7ae6-4c56-8878-e91f699f39f0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:30,635 [275] [DEBUG] [app] Ending request: urn:request:30edac6b-7ae6-4c56-8878-e91f699f39f0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30edac6b-7ae6-4c56-8878-e91f699f39f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:00:30,636 [280] [DEBUG] [app] Starting request: urn:request:6ad3c577-f863-4964-8ad0-bf8b849342f8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:00:30,636 [280] [DEBUG] [app] Ending request: urn:request:6ad3c577-f863-4964-8ad0-bf8b849342f8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ad3c577-f863-4964-8ad0-bf8b849342f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:00:30,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:30,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:30,638 [282] [DEBUG] [app] Starting request: urn:request:3de50b5c-7c69-4c6a-ac74-8ae769f9acbd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:30,638 [282] [DEBUG] [app] Ending request: urn:request:3de50b5c-7c69-4c6a-ac74-8ae769f9acbd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3de50b5c-7c69-4c6a-ac74-8ae769f9acbd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:00:30,639 [285] [DEBUG] [app] Starting request: urn:request:579be950-3c2f-42bb-ac6d-0413ec419844 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:30,639 [285] [DEBUG] [app] Ending request: urn:request:579be950-3c2f-42bb-ac6d-0413ec419844 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:579be950-3c2f-42bb-ac6d-0413ec419844', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:30,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:00:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:30,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:30,660 [282] [DEBUG] [app] Ending request: urn:request:960be804-5d1d-4bea-9d07-38bb94836158 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:960be804-5d1d-4bea-9d07-38bb94836158', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:00:30,660 [284] [DEBUG] [app] Ending request: urn:request:99907522-abbe-4da7-bd4a-547f595e602a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99907522-abbe-4da7-bd4a-547f595e602a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:30,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:00:30,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 04:00:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:00:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:00:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:01.842666+00:00 (in 29.999454 seconds)
gcworker stdout | 2024-04-22 04:00:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:01 UTC)" (scheduled at 2024-04-22 04:00:31.842666+00:00)
gcworker stdout | 2024-04-22 04:00:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548831843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:00:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:00:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:00:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:00:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:00:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:29.931002+00:00 (in 57.996808 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:00:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:31 UTC)" (scheduled at 2024-04-22 04:00:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:00:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:00:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:00:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:42.430973+00:00 (in 9.999527 seconds)
notificationworker stdout | 2024-04-22 04:00:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:42 UTC)" (scheduled at 2024-04-22 04:00:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:00:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:00:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 32, 431837), True, datetime.datetime(2024, 4, 22, 4, 0, 32, 431837), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:00:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:00:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:00:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:00:34,874 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:00:36,151 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:00:38,803 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:00:39,842 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:00:40,203 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:00:40,236 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
autopruneworker stdout | 2024-04-22 04:00:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:00:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:10.238173+00:00 (in 29.999527 seconds)
autopruneworker stdout | 2024-04-22 04:00:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:10 UTC)" (scheduled at 2024-04-22 04:00:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:00:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758380247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:00:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:00:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:00:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:10 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:00:40,898 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:00:41,516 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:00:41,734 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:00:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:52.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 04:00:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:52 UTC)" (scheduled at 2024-04-22 04:00:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:00:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:00:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 42, 431790), True, datetime.datetime(2024, 4, 22, 4, 0, 42, 431790), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:00:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:00:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:00:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:00:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:00:43,085 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:00:43,981 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:00:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:00:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:47.065745+00:00 (in 2.998199 seconds)
securityworker stdout | 2024-04-22 04:00:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:14 UTC)" (scheduled at 2024-04-22 04:00:44.067050+00:00)
securityworker stdout | 2024-04-22 04:00:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:00:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:00:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:00:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:00:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:00:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:00:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:00:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:00:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:00:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:00:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 55, 44, 77931), 1, 2])
securityworker stdout | 2024-04-22 04:00:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:00:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:00:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 55, 44, 77931), 1, 2])
securityworker stdout | 2024-04-22 04:00:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:00:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:00:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:00:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:00:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:00:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:00:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:00:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:00:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:00:45,632 [282] [DEBUG] [app] Starting request: urn:request:012fd1e2-355e-4e17-9b09-e418f5fb2cc8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:45,632 [286] [DEBUG] [app] Starting request: urn:request:3531d662-a763-4a88-80f9-b69b1607a012 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:00:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:00:45,635 [275] [DEBUG] [app] Starting request: urn:request:d35f6f68-1a64-482d-9ebe-211b1529ff71 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:45,635 [283] [DEBUG] [app] Starting request: urn:request:f0417952-4b13-496d-b139-1eb61a043621 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:00:45,636 [275] [DEBUG] [app] Ending request: urn:request:d35f6f68-1a64-482d-9ebe-211b1529ff71 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d35f6f68-1a64-482d-9ebe-211b1529ff71', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:00:45,636 [283] [DEBUG] [app] Ending request: urn:request:f0417952-4b13-496d-b139-1eb61a043621 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0417952-4b13-496d-b139-1eb61a043621', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:00:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:00:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:00:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:00:45,639 [285] [DEBUG] [app] Starting request: urn:request:5787680b-f700-442e-8c11-d40ad3816731 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:45,639 [282] [DEBUG] [app] Starting request: urn:request:29f2c18d-88b8-4a46-97c0-e592b7f0b77a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:00:45,639 [285] [DEBUG] [app] Ending request: urn:request:5787680b-f700-442e-8c11-d40ad3816731 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5787680b-f700-442e-8c11-d40ad3816731', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:45,639 [282] [DEBUG] [app] Ending request: urn:request:29f2c18d-88b8-4a46-97c0-e592b7f0b77a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:29f2c18d-88b8-4a46-97c0-e592b7f0b77a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:00:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:00:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:00:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:00:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:00:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:00:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:00:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:45,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:00:45,660 [286] [DEBUG] [app] Ending request: urn:request:3531d662-a763-4a88-80f9-b69b1607a012 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3531d662-a763-4a88-80f9-b69b1607a012', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 04:00:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:00:45,661 [282] [DEBUG] [app] Ending request: urn:request:012fd1e2-355e-4e17-9b09-e418f5fb2cc8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:012fd1e2-355e-4e17-9b09-e418f5fb2cc8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:00:45,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:00:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:00:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:00:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 04:00:46,439 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:00:46,698 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:00:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:00:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:14.067050+00:00 (in 27.000774 seconds)
securityworker stdout | 2024-04-22 04:00:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:17 UTC)" (scheduled at 2024-04-22 04:00:47.065745+00:00)
securityworker stdout | 2024-04-22 04:00:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:00:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:00:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:00:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:00:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:00:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:00:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:17 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:00:47,155 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
namespacegcworker stdout | 2024-04-22 04:00:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:00:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:10.248668+00:00 (in 23.002261 seconds)
namespacegcworker stdout | 2024-04-22 04:00:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:47 UTC)" (scheduled at 2024-04-22 04:00:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:00:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:00:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 47, 246673), True, datetime.datetime(2024, 4, 22, 4, 0, 47, 246673), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:00:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:00:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:00:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:00:47,831 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:00:48,764 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:00:49,751 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
securityworker stdout | 2024-04-22 04:00:49,754 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
gunicorn-secscan stdout | 2024-04-22 04:00:49,757 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:00:49,760 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
manifestbackfillworker stdout | 2024-04-22 04:00:50,230 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:00:51,248 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:00:51,316 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:00:51,324 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:00:51,330 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:00:51,337 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:00:51,340 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:00:51,352 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:00:51,358 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:00:51,361 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
blobuploadcleanupworker stdout | 2024-04-22 04:00:51,413 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-web stdout | 2024-04-22 04:00:51,440 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:00:51,444 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:00:51,447 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:00:51,449 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:00:51,452 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 04:00:51,733 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:00:51,934 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:00:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:00:54.433683+00:00 (in 2.002171 seconds)
notificationworker stdout | 2024-04-22 04:00:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:02 UTC)" (scheduled at 2024-04-22 04:00:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:00:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:00:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 52, 431804), True, datetime.datetime(2024, 4, 22, 4, 0, 52, 431804), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:00:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:00:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:00:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:00:52,758 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:00:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:00:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:18.047111+00:00 (in 25.001085 seconds)
exportactionlogsworker stdout | 2024-04-22 04:00:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:53 UTC)" (scheduled at 2024-04-22 04:00:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:00:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:00:53,106 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 0, 53, 46290), True, datetime.datetime(2024, 4, 22, 4, 0, 53, 46290), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:00:53,120 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:00:53,120 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:00:53,120 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:00:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:00:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:02.430973+00:00 (in 7.996754 seconds)
notificationworker stdout | 2024-04-22 04:00:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:54 UTC)" (scheduled at 2024-04-22 04:00:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:00:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:00:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:01:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:00:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:00:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:28.130973+00:00 (in 29.999514 seconds)
buildlogsarchiver stdout | 2024-04-22 04:00:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:28 UTC)" (scheduled at 2024-04-22 04:00:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:00:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 0, 58, 131689), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:00:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:00:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:00:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:01:00,632 [285] [DEBUG] [app] Starting request: urn:request:eccdf09a-699d-4c20-84fb-9d52ca613a18 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:00,632 [286] [DEBUG] [app] Starting request: urn:request:7ec13b14-31cf-4921-af8e-4918e1cfd2a3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:00,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:01:00,635 [280] [DEBUG] [app] Starting request: urn:request:cb761429-5b4c-465b-842b-b6a096d87dc6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:00,635 [279] [DEBUG] [app] Starting request: urn:request:7b90c837-baa9-4776-8383-38cfcfdd4313 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:00,636 [280] [DEBUG] [app] Ending request: urn:request:cb761429-5b4c-465b-842b-b6a096d87dc6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb761429-5b4c-465b-842b-b6a096d87dc6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:00,636 [279] [DEBUG] [app] Ending request: urn:request:7b90c837-baa9-4776-8383-38cfcfdd4313 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7b90c837-baa9-4776-8383-38cfcfdd4313', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:00,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:01:00,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:00,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:00,639 [284] [DEBUG] [app] Starting request: urn:request:429392e5-f4d9-4c60-8ccf-cb1023dc90be (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:00,639 [284] [DEBUG] [app] Ending request: urn:request:429392e5-f4d9-4c60-8ccf-cb1023dc90be (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:429392e5-f4d9-4c60-8ccf-cb1023dc90be', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:00,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:00,640 [285] [DEBUG] [app] Starting request: urn:request:c81f0f0d-6b34-457e-a492-5acb1477ea36 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:00,640 [285] [DEBUG] [app] Ending request: urn:request:c81f0f0d-6b34-457e-a492-5acb1477ea36 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c81f0f0d-6b34-457e-a492-5acb1477ea36', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:00,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:00,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:00,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:00,661 [285] [DEBUG] [app] Ending request: urn:request:eccdf09a-699d-4c20-84fb-9d52ca613a18 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eccdf09a-699d-4c20-84fb-9d52ca613a18', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:01:00,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:01:00,662 [286] [DEBUG] [app] Ending request: urn:request:7ec13b14-31cf-4921-af8e-4918e1cfd2a3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7ec13b14-31cf-4921-af8e-4918e1cfd2a3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 04:01:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 04:01:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:01:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:31.842666+00:00 (in 29.999424 seconds)
gcworker stdout | 2024-04-22 04:01:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:31 UTC)" (scheduled at 2024-04-22 04:01:01.842666+00:00)
gcworker stdout | 2024-04-22 04:01:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548861843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:01:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:01:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:01:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:01:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:01:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:09.033206+00:00 (in 6.997154 seconds)
repositorygcworker stdout | 2024-04-22 04:01:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:02 UTC)" (scheduled at 2024-04-22 04:01:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:01:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:01:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:01:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:12.430973+00:00 (in 9.999488 seconds)
notificationworker stdout | 2024-04-22 04:01:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:12 UTC)" (scheduled at 2024-04-22 04:01:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:01:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:01:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 2, 431769), True, datetime.datetime(2024, 4, 22, 4, 1, 2, 431769), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:01:02,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:01:02,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:01:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:01:04,909 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:01:06,187 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:01:08,840 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 04:01:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:01:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:02.035613+00:00 (in 53.001885 seconds)
repositorygcworker stdout | 2024-04-22 04:01:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:09 UTC)" (scheduled at 2024-04-22 04:01:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:01:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:01:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 9, 34185), True, datetime.datetime(2024, 4, 22, 4, 1, 9, 34185), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:01:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:01:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:01:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:01:09,874 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:01:10,221 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
autopruneworker stdout | 2024-04-22 04:01:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:01:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:40.238173+00:00 (in 29.999500 seconds)
autopruneworker stdout | 2024-04-22 04:01:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:40 UTC)" (scheduled at 2024-04-22 04:01:10.238173+00:00)
namespacegcworker stdout | 2024-04-22 04:01:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:01:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:30.247852+00:00 (in 19.998731 seconds)
namespacegcworker stdout | 2024-04-22 04:01:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:10 UTC)" (scheduled at 2024-04-22 04:01:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:01:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:01:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:01:10,254 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758410254, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:01:10,260 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:01:10,260 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:01:10,260 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:40 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 04:01:10,269 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:01:10,917 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:01:11,548 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:01:11,767 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:01:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:22.430973+00:00 (in 9.999475 seconds)
notificationworker stdout | 2024-04-22 04:01:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:22 UTC)" (scheduled at 2024-04-22 04:01:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:01:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:01:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 12, 431956), True, datetime.datetime(2024, 4, 22, 4, 1, 12, 431956), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:01:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:01:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:01:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:01:13,116 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:01:13,998 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:01:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:01:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:17.065745+00:00 (in 2.998199 seconds)
securityworker stdout | 2024-04-22 04:01:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:44 UTC)" (scheduled at 2024-04-22 04:01:14.067050+00:00)
securityworker stdout | 2024-04-22 04:01:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:01:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:01:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:01:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:01:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:01:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:01:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:01:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:01:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:01:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:01:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 56, 14, 77292), 1, 2])
securityworker stdout | 2024-04-22 04:01:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:01:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:01:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 56, 14, 77292), 1, 2])
securityworker stdout | 2024-04-22 04:01:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:01:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:01:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:01:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:01:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:01:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:01:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:01:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:01:15,632 [286] [DEBUG] [app] Starting request: urn:request:c37cb607-03a2-4826-ad19-b70ae3ecab99 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:15,632 [282] [DEBUG] [app] Starting request: urn:request:47e45bb8-b401-4be2-968c-4308a8e84baa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:01:15,635 [283] [DEBUG] [app] Starting request: urn:request:ddec289c-f3ee-4688-bf87-5a332494257e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:15,635 [283] [DEBUG] [app] Ending request: urn:request:ddec289c-f3ee-4688-bf87-5a332494257e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ddec289c-f3ee-4688-bf87-5a332494257e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:15,636 [275] [DEBUG] [app] Starting request: urn:request:66fe4b36-62fc-410e-824a-7b5b3dbb9f4a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:01:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 04:01:15,636 [275] [DEBUG] [app] Ending request: urn:request:66fe4b36-62fc-410e-824a-7b5b3dbb9f4a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:66fe4b36-62fc-410e-824a-7b5b3dbb9f4a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:01:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:15,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:15,639 [285] [DEBUG] [app] Starting request: urn:request:2f2aef4d-7817-475c-b3f2-a5ec1b771ed9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:15,639 [285] [DEBUG] [app] Ending request: urn:request:2f2aef4d-7817-475c-b3f2-a5ec1b771ed9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2f2aef4d-7817-475c-b3f2-a5ec1b771ed9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:01:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:15,640 [284] [DEBUG] [app] Starting request: urn:request:09624a46-f0cf-4047-b6ac-48c65e8e15cf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:15,640 [284] [DEBUG] [app] Ending request: urn:request:09624a46-f0cf-4047-b6ac-48c65e8e15cf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:09624a46-f0cf-4047-b6ac-48c65e8e15cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:01:15,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:15,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:15,661 [282] [DEBUG] [app] Ending request: urn:request:47e45bb8-b401-4be2-968c-4308a8e84baa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:47e45bb8-b401-4be2-968c-4308a8e84baa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:01:15,661 [286] [DEBUG] [app] Ending request: urn:request:c37cb607-03a2-4826-ad19-b70ae3ecab99 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c37cb607-03a2-4826-ad19-b70ae3ecab99', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:15,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:01:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:01:15,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.031)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
expiredappspecifictokenworker stdout | 2024-04-22 04:01:16,476 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:01:16,726 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:01:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:01:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:44.067050+00:00 (in 27.000800 seconds)
securityworker stdout | 2024-04-22 04:01:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:47 UTC)" (scheduled at 2024-04-22 04:01:17.065745+00:00)
securityworker stdout | 2024-04-22 04:01:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:01:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:01:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:01:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:01:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:01:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:01:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:01:17,192 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:01:17,854 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 04:01:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:01:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:53.045554+00:00 (in 34.997924 seconds)
exportactionlogsworker stdout | 2024-04-22 04:01:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:18 UTC)" (scheduled at 2024-04-22 04:01:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:01:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:01:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:01:18,797 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:01:19,760 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:01:19,764 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:01:19,767 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:01:19,789 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:01:20,243 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:01:21,257 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:01:21,325 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:01:21,331 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:01:21,336 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:01:21,348 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:01:21,351 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:01:21,361 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:01:21,364 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:01:21,368 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:01:21,448 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
blobuploadcleanupworker stdout | 2024-04-22 04:01:21,451 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
gunicorn-web stdout | 2024-04-22 04:01:21,455 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:01:21,458 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:01:21,460 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:01:21,463 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
namespacegcworker stdout | 2024-04-22 04:01:21,769 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:01:21,971 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:01:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:32.430973+00:00 (in 9.999476 seconds)
notificationworker stdout | 2024-04-22 04:01:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:32 UTC)" (scheduled at 2024-04-22 04:01:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:01:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:01:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 22, 431776), True, datetime.datetime(2024, 4, 22, 4, 1, 22, 431776), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:01:22,451 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:01:22,451 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:01:22,451 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:01:22,772 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:01:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:01:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:24.738365+00:00 (in 59.999453 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:01:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:24 UTC)" (scheduled at 2024-04-22 04:01:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:01:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:01:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:01:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:01:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:01:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:58.130973+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2024-04-22 04:01:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:58 UTC)" (scheduled at 2024-04-22 04:01:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:01:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 1, 28, 131745), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:01:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:01:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:01:28,148 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:01:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:31.933721+00:00 (in 2.002181 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:29 UTC)" (scheduled at 2024-04-22 04:01:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 29, 931829), True, datetime.datetime(2024, 4, 22, 4, 1, 29, 931829), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,956 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,956 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:01:29,956 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:01:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:01:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:47.245913+00:00 (in 16.997527 seconds)
namespacegcworker stdout | 2024-04-22 04:01:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:06:30 UTC)" (scheduled at 2024-04-22 04:01:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 04:01:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 1, 30, 248588), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:01:30,261 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 1, 30, 248588), True, datetime.datetime(2024, 4, 22, 4, 1, 30, 248588), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:01:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 1, 30, 248588), True, datetime.datetime(2024, 4, 22, 4, 1, 30, 248588), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 4, 1, 30, 248588), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:01:30,270 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:01:30,270 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:06:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:01:30,632 [282] [DEBUG] [app] Starting request: urn:request:ac96b98f-2729-4d74-aaf6-c8ae616f20b3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:30,632 [286] [DEBUG] [app] Starting request: urn:request:dc06f9f4-f2f7-4eb5-9e47-b01178f51c69 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:01:30,635 [279] [DEBUG] [app] Starting request: urn:request:ceefe7e9-9dce-4a15-8fd8-a93fdc066ee5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:30,635 [275] [DEBUG] [app] Starting request: urn:request:ebb44128-74a1-4732-bd1e-6f7230accf59 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:30,636 [275] [DEBUG] [app] Ending request: urn:request:ebb44128-74a1-4732-bd1e-6f7230accf59 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ebb44128-74a1-4732-bd1e-6f7230accf59', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:30,636 [279] [DEBUG] [app] Ending request: urn:request:ceefe7e9-9dce-4a15-8fd8-a93fdc066ee5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ceefe7e9-9dce-4a15-8fd8-a93fdc066ee5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:01:30,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:01:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:30,638 [282] [DEBUG] [app] Starting request: urn:request:7aa932fa-800c-4c3c-935c-aad719903dd8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:30,639 [282] [DEBUG] [app] Ending request: urn:request:7aa932fa-800c-4c3c-935c-aad719903dd8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7aa932fa-800c-4c3c-935c-aad719903dd8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:01:30,639 [285] [DEBUG] [app] Starting request: urn:request:f556ea5e-ba2c-4521-b1dd-c01bc060d0d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:30,639 [285] [DEBUG] [app] Ending request: urn:request:f556ea5e-ba2c-4521-b1dd-c01bc060d0d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f556ea5e-ba2c-4521-b1dd-c01bc060d0d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:30,660 [286] [DEBUG] [app] Ending request: urn:request:dc06f9f4-f2f7-4eb5-9e47-b01178f51c69 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dc06f9f4-f2f7-4eb5-9e47-b01178f51c69', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:01:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:01:30,661 [282] [DEBUG] [app] Ending request: urn:request:ac96b98f-2729-4d74-aaf6-c8ae616f20b3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ac96b98f-2729-4d74-aaf6-c8ae616f20b3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:01:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:01:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:01:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:01:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:01.842666+00:00 (in 29.999462 seconds)
gcworker stdout | 2024-04-22 04:01:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:01 UTC)" (scheduled at 2024-04-22 04:01:31.842666+00:00)
gcworker stdout | 2024-04-22 04:01:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548891843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:01:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:01:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:01:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:01:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:01:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:29.931002+00:00 (in 57.996799 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:01:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:31 UTC)" (scheduled at 2024-04-22 04:01:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:01:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:01:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:01:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:42.430973+00:00 (in 9.999514 seconds)
notificationworker stdout | 2024-04-22 04:01:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:42 UTC)" (scheduled at 2024-04-22 04:01:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:01:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:01:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 32, 431693), True, datetime.datetime(2024, 4, 22, 4, 1, 32, 431693), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:01:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:01:32,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:01:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:01:34,945 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:01:36,223 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:01:38,849 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:01:39,911 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:01:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:01:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:10.238173+00:00 (in 29.999478 seconds)
autopruneworker stdout | 2024-04-22 04:01:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:10 UTC)" (scheduled at 2024-04-22 04:01:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:01:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758440248, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:01:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:01:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:01:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:01:40,258 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:01:40,304 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:01:40,954 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:01:41,577 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:01:41,802 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:01:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:52.430973+00:00 (in 9.999501 seconds)
notificationworker stdout | 2024-04-22 04:01:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:52 UTC)" (scheduled at 2024-04-22 04:01:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:01:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:01:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 42, 431839), True, datetime.datetime(2024, 4, 22, 4, 1, 42, 431839), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:01:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:01:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:01:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:01:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:01:43,143 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:01:44,035 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:01:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:01:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:47.065745+00:00 (in 2.998152 seconds)
securityworker stdout | 2024-04-22 04:01:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:14 UTC)" (scheduled at 2024-04-22 04:01:44.067050+00:00)
securityworker stdout | 2024-04-22 04:01:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:01:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:01:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:01:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:01:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:01:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:01:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:01:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:01:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:01:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:01:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 56, 44, 77594), 1, 2])
securityworker stdout | 2024-04-22 04:01:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:01:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:01:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 56, 44, 77594), 1, 2])
securityworker stdout | 2024-04-22 04:01:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:01:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:01:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:01:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:01:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:01:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:01:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:01:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:01:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:01:45,632 [282] [DEBUG] [app] Starting request: urn:request:bbd853df-4acf-4395-8421-f3e91a00bbad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:45,632 [284] [DEBUG] [app] Starting request: urn:request:bdc8cff0-099b-4874-a8e5-673267a8b13b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:01:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:01:45,635 [283] [DEBUG] [app] Starting request: urn:request:4d4263a6-fdcc-4644-81eb-a1e4583fd393 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:45,635 [281] [DEBUG] [app] Starting request: urn:request:b2da4408-eb54-45f3-927d-9c7aee974676 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:01:45,636 [283] [DEBUG] [app] Ending request: urn:request:4d4263a6-fdcc-4644-81eb-a1e4583fd393 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4d4263a6-fdcc-4644-81eb-a1e4583fd393', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:45,636 [281] [DEBUG] [app] Ending request: urn:request:b2da4408-eb54-45f3-927d-9c7aee974676 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2da4408-eb54-45f3-927d-9c7aee974676', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:01:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:01:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:01:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:01:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:01:45,639 [285] [DEBUG] [app] Starting request: urn:request:da64de5e-4ebc-46b8-8fa4-fcf5a8bfb56d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:45,639 [285] [DEBUG] [app] Ending request: urn:request:da64de5e-4ebc-46b8-8fa4-fcf5a8bfb56d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:da64de5e-4ebc-46b8-8fa4-fcf5a8bfb56d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:01:45,639 [286] [DEBUG] [app] Starting request: urn:request:3fc61ab6-844b-4e6d-88c2-a79233089454 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:01:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:01:45,639 [286] [DEBUG] [app] Ending request: urn:request:3fc61ab6-844b-4e6d-88c2-a79233089454 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3fc61ab6-844b-4e6d-88c2-a79233089454', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:01:45,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:01:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:01:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:01:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:01:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:01:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:01:45,660 [284] [DEBUG] [app] Ending request: urn:request:bdc8cff0-099b-4874-a8e5-673267a8b13b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bdc8cff0-099b-4874-a8e5-673267a8b13b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:45,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:01:45,660 [282] [DEBUG] [app] Ending request: urn:request:bbd853df-4acf-4395-8421-f3e91a00bbad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bbd853df-4acf-4395-8421-f3e91a00bbad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:01:45,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:01:45,661 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:01:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:01:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:01:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 04:01:46,509 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:01:46,763 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:01:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:01:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:14.067050+00:00 (in 27.000743 seconds)
securityworker stdout | 2024-04-22 04:01:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:17 UTC)" (scheduled at 2024-04-22 04:01:47.065745+00:00)
securityworker stdout | 2024-04-22 04:01:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:01:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:01:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:01:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:01:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:01:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:01:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:17 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:01:47,229 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
namespacegcworker stdout | 2024-04-22 04:01:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:01:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:10.248668+00:00 (in 23.002303 seconds)
namespacegcworker stdout | 2024-04-22 04:01:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:47 UTC)" (scheduled at 2024-04-22 04:01:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:01:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:01:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 47, 246604), True, datetime.datetime(2024, 4, 22, 4, 1, 47, 246604), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:01:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:01:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:01:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:01:47,865 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:01:48,815 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:01:49,769 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:01:49,772 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:01:49,775 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:01:49,825 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:01:50,280 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:01:51,269 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:01:51,332 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:01:51,339 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:01:51,347 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:01:51,357 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:01:51,359 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:01:51,367 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:01:51,372 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:01:51,374 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:01:51,456 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:01:51,462 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:01:51,466 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:01:51,469 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:01:51,471 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:01:51,487 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:01:51,805 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:01:51,979 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:01:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:01:54.433683+00:00 (in 2.002197 seconds)
notificationworker stdout | 2024-04-22 04:01:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:02 UTC)" (scheduled at 2024-04-22 04:01:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:01:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:01:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 52, 431762), True, datetime.datetime(2024, 4, 22, 4, 1, 52, 431762), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:01:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:01:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:01:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:01:52,791 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:01:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:01:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:18.047111+00:00 (in 25.001090 seconds)
exportactionlogsworker stdout | 2024-04-22 04:01:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:53 UTC)" (scheduled at 2024-04-22 04:01:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:01:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:01:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 1, 53, 46339), True, datetime.datetime(2024, 4, 22, 4, 1, 53, 46339), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:01:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:01:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:01:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:01:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:01:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:02.430973+00:00 (in 7.996825 seconds)
notificationworker stdout | 2024-04-22 04:01:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:54 UTC)" (scheduled at 2024-04-22 04:01:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:01:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:01:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:02:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:01:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:01:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:28.130973+00:00 (in 29.999524 seconds)
buildlogsarchiver stdout | 2024-04-22 04:01:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:28 UTC)" (scheduled at 2024-04-22 04:01:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:01:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 1, 58, 131819), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:01:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:01:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:01:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:02:00,632 [282] [DEBUG] [app] Starting request: urn:request:3f1d9d71-5982-4c51-8f15-1a30c4a670c0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:00,632 [285] [DEBUG] [app] Starting request: urn:request:0d0cb792-dbe6-4faf-8048-fbbdf71dca13 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:02:00,636 [283] [DEBUG] [app] Starting request: urn:request:2407a712-ec4f-4fb2-9faa-119f3ff0a100 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:00,636 [275] [DEBUG] [app] Starting request: urn:request:e16d4b90-a043-4568-859d-d544f62fe3c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:00,636 [283] [DEBUG] [app] Ending request: urn:request:2407a712-ec4f-4fb2-9faa-119f3ff0a100 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2407a712-ec4f-4fb2-9faa-119f3ff0a100', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:00,636 [275] [DEBUG] [app] Ending request: urn:request:e16d4b90-a043-4568-859d-d544f62fe3c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e16d4b90-a043-4568-859d-d544f62fe3c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:02:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:02:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:00,639 [285] [DEBUG] [app] Starting request: urn:request:548c7190-a339-47d7-9c10-50a43c9d98ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:00,639 [284] [DEBUG] [app] Starting request: urn:request:ffca24b0-777d-4519-bf68-f3171f1960f8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:00,639 [285] [DEBUG] [app] Ending request: urn:request:548c7190-a339-47d7-9c10-50a43c9d98ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:548c7190-a339-47d7-9c10-50a43c9d98ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:00,639 [284] [DEBUG] [app] Ending request: urn:request:ffca24b0-777d-4519-bf68-f3171f1960f8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ffca24b0-777d-4519-bf68-f3171f1960f8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:02:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:00,661 [285] [DEBUG] [app] Ending request: urn:request:0d0cb792-dbe6-4faf-8048-fbbdf71dca13 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d0cb792-dbe6-4faf-8048-fbbdf71dca13', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:02:00,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:02:00,662 [282] [DEBUG] [app] Ending request: urn:request:3f1d9d71-5982-4c51-8f15-1a30c4a670c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3f1d9d71-5982-4c51-8f15-1a30c4a670c0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 04:02:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:02:00,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:02:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:02:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:31.842666+00:00 (in 29.999475 seconds)
gcworker stdout | 2024-04-22 04:02:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:31 UTC)" (scheduled at 2024-04-22 04:02:01.842666+00:00)
gcworker stdout | 2024-04-22 04:02:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548921843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:02:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:02:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:02:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:02:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:02:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:09.033206+00:00 (in 6.997081 seconds)
repositorygcworker stdout | 2024-04-22 04:02:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:02 UTC)" (scheduled at 2024-04-22 04:02:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:02:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:02:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:02:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:12.430973+00:00 (in 9.999531 seconds)
notificationworker stdout | 2024-04-22 04:02:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:12 UTC)" (scheduled at 2024-04-22 04:02:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:02:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:02:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 2, 431868), True, datetime.datetime(2024, 4, 22, 4, 2, 2, 431868), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:02:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:02:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:02:04,965 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:02:06,260 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:02:08,885 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 04:02:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:02:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:02.035613+00:00 (in 53.001902 seconds)
repositorygcworker stdout | 2024-04-22 04:02:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:09 UTC)" (scheduled at 2024-04-22 04:02:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:02:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:02:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 9, 34173), True, datetime.datetime(2024, 4, 22, 4, 2, 9, 34173), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:02:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:02:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:02:09,050 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:02:09,947 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:02:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:02:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:40.238173+00:00 (in 29.999542 seconds)
autopruneworker stdout | 2024-04-22 04:02:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:40 UTC)" (scheduled at 2024-04-22 04:02:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:02:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758470248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:02:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:02:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:47.245913+00:00 (in 36.996762 seconds)
namespacegcworker stdout | 2024-04-22 04:02:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:10 UTC)" (scheduled at 2024-04-22 04:02:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:02:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:02:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:02:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:02:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:02:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:02:10,283 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:02:10,334 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:02:10,968 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:02:11,614 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:02:11,838 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:02:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:22.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 04:02:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:22 UTC)" (scheduled at 2024-04-22 04:02:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:02:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:02:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 12, 431794), True, datetime.datetime(2024, 4, 22, 4, 2, 12, 431794), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:02:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:02:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:02:13,149 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:02:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:02:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:17.065745+00:00 (in 2.998140 seconds)
securityworker stdout | 2024-04-22 04:02:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:44 UTC)" (scheduled at 2024-04-22 04:02:14.067050+00:00)
securityworker stdout | 2024-04-22 04:02:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:02:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:02:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:02:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:02:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
repositorygcworker stdout | 2024-04-22 04:02:14,072 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:02:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:02:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:02:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:02:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:02:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:02:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 57, 14, 77292), 1, 2])
securityworker stdout | 2024-04-22 04:02:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:02:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:02:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 57, 14, 77292), 1, 2])
securityworker stdout | 2024-04-22 04:02:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:02:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:02:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:02:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:02:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:02:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:02:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:02:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:02:15,632 [285] [DEBUG] [app] Starting request: urn:request:f97bfa60-f17e-46cd-ae45-46282c228c58 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:15,632 [284] [DEBUG] [app] Starting request: urn:request:e5669811-5b0e-4338-ac0f-97a06b81a206 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:02:15,635 [275] [DEBUG] [app] Starting request: urn:request:d4e74d20-957c-4ebd-b63e-6c37d4f6a5ed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:15,636 [276] [DEBUG] [app] Starting request: urn:request:2ca14d3a-bc84-4624-ab2a-4ff98e871a58 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:15,636 [276] [DEBUG] [app] Ending request: urn:request:2ca14d3a-bc84-4624-ab2a-4ff98e871a58 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ca14d3a-bc84-4624-ab2a-4ff98e871a58', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:15,636 [275] [DEBUG] [app] Ending request: urn:request:d4e74d20-957c-4ebd-b63e-6c37d4f6a5ed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d4e74d20-957c-4ebd-b63e-6c37d4f6a5ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:15,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:02:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:02:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:15,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:15,639 [282] [DEBUG] [app] Starting request: urn:request:83c5beb0-46fb-4eb3-8f91-cc65efa703cd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:15,639 [282] [DEBUG] [app] Ending request: urn:request:83c5beb0-46fb-4eb3-8f91-cc65efa703cd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:83c5beb0-46fb-4eb3-8f91-cc65efa703cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:15,639 [286] [DEBUG] [app] Starting request: urn:request:b8ace231-6f33-49b0-a129-d3be4b233115 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:15,639 [286] [DEBUG] [app] Ending request: urn:request:b8ace231-6f33-49b0-a129-d3be4b233115 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b8ace231-6f33-49b0-a129-d3be4b233115', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:02:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:15,660 [285] [DEBUG] [app] Ending request: urn:request:f97bfa60-f17e-46cd-ae45-46282c228c58 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f97bfa60-f17e-46cd-ae45-46282c228c58', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:15,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:02:15,661 [284] [DEBUG] [app] Ending request: urn:request:e5669811-5b0e-4338-ac0f-97a06b81a206 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e5669811-5b0e-4338-ac0f-97a06b81a206', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:15,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:02:15,661 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:02:15,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:02:16,522 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:02:16,784 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:02:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:02:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:44.067050+00:00 (in 27.000823 seconds)
securityworker stdout | 2024-04-22 04:02:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:47 UTC)" (scheduled at 2024-04-22 04:02:17.065745+00:00)
securityworker stdout | 2024-04-22 04:02:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:02:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:02:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:02:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:02:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:02:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:02:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:02:17,239 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:02:17,888 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 04:02:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:02:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:23.046900+00:00 (in 4.999297 seconds)
exportactionlogsworker stdout | 2024-04-22 04:02:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:18 UTC)" (scheduled at 2024-04-22 04:02:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:02:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:02:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:02:18,851 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:02:19,779 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:02:19,782 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:02:19,784 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:02:19,860 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:02:20,316 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:02:21,281 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:02:21,340 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:02:21,351 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:02:21,357 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:02:21,367 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:02:21,371 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:02:21,379 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:02:21,383 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:02:21,386 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:02:21,463 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:02:21,472 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:02:21,474 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:02:21,477 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:02:21,480 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:02:21,516 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:02:21,841 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:02:22,015 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:02:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:32.430973+00:00 (in 9.999467 seconds)
notificationworker stdout | 2024-04-22 04:02:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:32 UTC)" (scheduled at 2024-04-22 04:02:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:02:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:02:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 22, 431799), True, datetime.datetime(2024, 4, 22, 4, 2, 22, 431799), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:02:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:02:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:02:22,825 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:02:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:02:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:53.045554+00:00 (in 29.998120 seconds)
exportactionlogsworker stdout | 2024-04-22 04:02:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:07:23 UTC)" (scheduled at 2024-04-22 04:02:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 04:02:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 2, 23, 47685), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:02:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 2, 23, 47685), True, datetime.datetime(2024, 4, 22, 4, 2, 23, 47685), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:02:23,066 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 2, 23, 47685), True, datetime.datetime(2024, 4, 22, 4, 2, 23, 47685), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 4, 2, 23, 47685), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:02:23,070 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:02:23,070 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:07:23 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 04:02:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:02:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:24.738365+00:00 (in 59.999496 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:02:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:24 UTC)" (scheduled at 2024-04-22 04:02:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:02:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:02:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:02:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:24 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:02:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:02:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:58.130973+00:00 (in 29.999514 seconds)
buildlogsarchiver stdout | 2024-04-22 04:02:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:58 UTC)" (scheduled at 2024-04-22 04:02:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:02:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 2, 28, 131837), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:02:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:02:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:02:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:02:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:31.933721+00:00 (in 2.002158 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:29 UTC)" (scheduled at 2024-04-22 04:02:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 29, 931894), True, datetime.datetime(2024, 4, 22, 4, 2, 29, 931894), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:02:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:02:30,632 [286] [DEBUG] [app] Starting request: urn:request:8e53b014-9562-47d6-8277-9660fe20828d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:30,632 [282] [DEBUG] [app] Starting request: urn:request:110d9e02-7c8a-4e86-a079-bf8fc4a92c12 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:02:30,635 [281] [DEBUG] [app] Starting request: urn:request:24838353-eb9b-4be3-9de7-ded1e97d2aed (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:30,635 [283] [DEBUG] [app] Starting request: urn:request:b96d6521-edce-4f85-acba-ddee02f56d65 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:30,636 [281] [DEBUG] [app] Ending request: urn:request:24838353-eb9b-4be3-9de7-ded1e97d2aed (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24838353-eb9b-4be3-9de7-ded1e97d2aed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:30,636 [283] [DEBUG] [app] Ending request: urn:request:b96d6521-edce-4f85-acba-ddee02f56d65 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b96d6521-edce-4f85-acba-ddee02f56d65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:02:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:02:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:02:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:30,639 [284] [DEBUG] [app] Starting request: urn:request:7607b90f-f188-40d6-b047-44389f72d72f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:30,639 [285] [DEBUG] [app] Starting request: urn:request:8e3725cb-9d4d-4f18-81b0-68cc1ac30a1f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:30,640 [285] [DEBUG] [app] Ending request: urn:request:8e3725cb-9d4d-4f18-81b0-68cc1ac30a1f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8e3725cb-9d4d-4f18-81b0-68cc1ac30a1f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:30,640 [284] [DEBUG] [app] Ending request: urn:request:7607b90f-f188-40d6-b047-44389f72d72f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7607b90f-f188-40d6-b047-44389f72d72f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:02:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:30,662 [286] [DEBUG] [app] Ending request: urn:request:8e53b014-9562-47d6-8277-9660fe20828d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8e53b014-9562-47d6-8277-9660fe20828d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:30,663 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:02:30,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 04:02:30,674 [282] [DEBUG] [app] Ending request: urn:request:110d9e02-7c8a-4e86-a079-bf8fc4a92c12 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:110d9e02-7c8a-4e86-a079-bf8fc4a92c12', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:30,675 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:02:30,675 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.044 118 0.044)
gcworker stdout | 2024-04-22 04:02:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:02:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:01.842666+00:00 (in 29.999472 seconds)
gcworker stdout | 2024-04-22 04:02:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:01 UTC)" (scheduled at 2024-04-22 04:02:31.842666+00:00)
gcworker stdout | 2024-04-22 04:02:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548951843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:02:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:02:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:02:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:02:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:02:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:29.931002+00:00 (in 57.996786 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:02:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:31 UTC)" (scheduled at 2024-04-22 04:02:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:02:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:02:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:02:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:38.433394+00:00 (in 6.001892 seconds)
notificationworker stdout | 2024-04-22 04:02:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:42 UTC)" (scheduled at 2024-04-22 04:02:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:02:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:02:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 32, 431935), True, datetime.datetime(2024, 4, 22, 4, 2, 32, 431935), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:02:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:02:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:02:35,001 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:02:36,296 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
notificationworker stdout | 2024-04-22 04:02:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:42.430973+00:00 (in 3.997106 seconds)
notificationworker stdout | 2024-04-22 04:02:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:07:38 UTC)" (scheduled at 2024-04-22 04:02:38.433394+00:00)
notificationworker stdout | 2024-04-22 04:02:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 2, 38, 434231), 'notification/%'])
notificationworker stdout | 2024-04-22 04:02:38,447 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 2, 38, 434231), True, datetime.datetime(2024, 4, 22, 4, 2, 38, 434231), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 04:02:38,451 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 2, 38, 434231), True, datetime.datetime(2024, 4, 22, 4, 2, 38, 434231), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 4, 2, 38, 434231), 'notification/%'])
notificationworker stdout | 2024-04-22 04:02:38,455 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:38,455 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:07:38 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:02:38,919 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:02:39,975 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:02:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:02:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:10.238173+00:00 (in 29.999534 seconds)
autopruneworker stdout | 2024-04-22 04:02:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:10 UTC)" (scheduled at 2024-04-22 04:02:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:02:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758500248, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:02:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:02:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:02:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:02:40,317 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:02:40,360 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:02:40,975 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:02:41,635 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:02:41,875 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:02:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:52.430973+00:00 (in 9.999518 seconds)
notificationworker stdout | 2024-04-22 04:02:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:52 UTC)" (scheduled at 2024-04-22 04:02:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:02:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:02:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 42, 431866), True, datetime.datetime(2024, 4, 22, 4, 2, 42, 431866), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:02:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:02:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:02:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:02:43,185 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:02:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:02:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:47.065745+00:00 (in 2.998155 seconds)
securityworker stdout | 2024-04-22 04:02:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:14 UTC)" (scheduled at 2024-04-22 04:02:44.067050+00:00)
securityworker stdout | 2024-04-22 04:02:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:02:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:02:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:02:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:02:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:02:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:02:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:02:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:02:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:02:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:02:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 57, 44, 77771), 1, 2])
repositorygcworker stdout | 2024-04-22 04:02:44,096 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:02:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:02:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:02:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 57, 44, 77771), 1, 2])
securityworker stdout | 2024-04-22 04:02:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:02:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:02:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:02:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:02:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:02:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:02:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:02:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:02:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:14 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:02:45,633 [282] [DEBUG] [app] Starting request: urn:request:c8ee8822-ec6f-4d55-8797-e3be75fbcf0d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:45,633 [286] [DEBUG] [app] Starting request: urn:request:cf6079ac-21a6-4410-aa43-2e02bbf6ad40 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:02:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:02:45,636 [278] [DEBUG] [app] Starting request: urn:request:26b0fa17-4540-4e37-b878-88802f4cb87b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:45,636 [275] [DEBUG] [app] Starting request: urn:request:b01fbb78-0f42-41c5-8b7e-a38e440d13c6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:02:45,636 [278] [DEBUG] [app] Ending request: urn:request:26b0fa17-4540-4e37-b878-88802f4cb87b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:26b0fa17-4540-4e37-b878-88802f4cb87b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:45,636 [275] [DEBUG] [app] Ending request: urn:request:b01fbb78-0f42-41c5-8b7e-a38e440d13c6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b01fbb78-0f42-41c5-8b7e-a38e440d13c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:02:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:02:45,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:02:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:45,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:02:45,639 [286] [DEBUG] [app] Starting request: urn:request:99168a31-9c84-408b-8543-3a382ff48638 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:45,639 [285] [DEBUG] [app] Starting request: urn:request:c9e9a207-8d02-4313-9373-ae204b75e21f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:02:45,639 [286] [DEBUG] [app] Ending request: urn:request:99168a31-9c84-408b-8543-3a382ff48638 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:99168a31-9c84-408b-8543-3a382ff48638', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:45,639 [285] [DEBUG] [app] Ending request: urn:request:c9e9a207-8d02-4313-9373-ae204b75e21f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c9e9a207-8d02-4313-9373-ae204b75e21f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:02:45,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.000)
gunicorn-web stdout | 2024-04-22 04:02:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:02:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:02:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:02:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:02:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:02:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:02:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:02:45,659 [286] [DEBUG] [app] Ending request: urn:request:cf6079ac-21a6-4410-aa43-2e02bbf6ad40 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf6079ac-21a6-4410-aa43-2e02bbf6ad40', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:02:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:02:45,660 [282] [DEBUG] [app] Ending request: urn:request:c8ee8822-ec6f-4d55-8797-e3be75fbcf0d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c8ee8822-ec6f-4d55-8797-e3be75fbcf0d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:02:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:02:45,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:02:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:02:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
expiredappspecifictokenworker stdout | 2024-04-22 04:02:46,557 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:02:46,821 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:02:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:02:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:14.067050+00:00 (in 27.000737 seconds)
securityworker stdout | 2024-04-22 04:02:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:17 UTC)" (scheduled at 2024-04-22 04:02:47.065745+00:00)
securityworker stdout | 2024-04-22 04:02:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:02:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:02:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:02:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:02:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:02:47,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:02:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:02:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:02:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:10.248668+00:00 (in 23.002318 seconds)
namespacegcworker stdout | 2024-04-22 04:02:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:47 UTC)" (scheduled at 2024-04-22 04:02:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:02:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
globalpromstats stdout | 2024-04-22 04:02:47,246 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
namespacegcworker stdout | 2024-04-22 04:02:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 47, 246596), True, datetime.datetime(2024, 4, 22, 4, 2, 47, 246596), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:02:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:02:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:02:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:47 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:02:47,925 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:02:48,887 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:02:49,788 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:02:49,792 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:02:49,794 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:02:49,869 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:02:50,349 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:02:51,292 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:02:51,353 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:02:51,362 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:02:51,367 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:02:51,377 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:02:51,379 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:02:51,390 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:02:51,394 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:02:51,397 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:02:51,470 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:02:51,479 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:02:51,482 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:02:51,485 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:02:51,488 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:02:51,551 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:02:51,875 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:02:52,045 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:02:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:02:54.433683+00:00 (in 2.002206 seconds)
notificationworker stdout | 2024-04-22 04:02:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:02 UTC)" (scheduled at 2024-04-22 04:02:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:02:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:02:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 52, 431926), True, datetime.datetime(2024, 4, 22, 4, 2, 52, 431926), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:02:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:02:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:02:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:02 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:02:52,854 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:02:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:02:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:18.047111+00:00 (in 25.000992 seconds)
exportactionlogsworker stdout | 2024-04-22 04:02:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:53 UTC)" (scheduled at 2024-04-22 04:02:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:02:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:02:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 2, 53, 46408), True, datetime.datetime(2024, 4, 22, 4, 2, 53, 46408), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:02:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:02:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:02:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:02:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:02:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:02.430973+00:00 (in 7.996811 seconds)
notificationworker stdout | 2024-04-22 04:02:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:54 UTC)" (scheduled at 2024-04-22 04:02:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:02:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:02:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:03:54 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:02:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:02:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:28.130973+00:00 (in 29.999504 seconds)
buildlogsarchiver stdout | 2024-04-22 04:02:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:28 UTC)" (scheduled at 2024-04-22 04:02:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:02:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 2, 58, 131695), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:02:58,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:02:58,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:02:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:03:00,633 [286] [DEBUG] [app] Starting request: urn:request:e4290c3f-dc06-4e4a-b6e3-fdaf162cfe66 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:00,633 [282] [DEBUG] [app] Starting request: urn:request:9f8c805c-e462-43ed-93ac-fefae240ae15 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:03:00,636 [283] [DEBUG] [app] Starting request: urn:request:1c0e0a9b-d0da-4082-8a0e-3c4f398eb9f4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:00,636 [278] [DEBUG] [app] Starting request: urn:request:f5265cd4-1ba8-4632-b04e-ad87cc5d4eef (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:00,636 [283] [DEBUG] [app] Ending request: urn:request:1c0e0a9b-d0da-4082-8a0e-3c4f398eb9f4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c0e0a9b-d0da-4082-8a0e-3c4f398eb9f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:03:00,636 [278] [DEBUG] [app] Ending request: urn:request:f5265cd4-1ba8-4632-b04e-ad87cc5d4eef (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f5265cd4-1ba8-4632-b04e-ad87cc5d4eef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:03:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:03:00,637 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:00,639 [286] [DEBUG] [app] Starting request: urn:request:4fab65a0-bee5-4a53-ae9a-c6ca8bdcf4a7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:03:00,640 [286] [DEBUG] [app] Ending request: urn:request:4fab65a0-bee5-4a53-ae9a-c6ca8bdcf4a7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4fab65a0-bee5-4a53-ae9a-c6ca8bdcf4a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:00,640 [284] [DEBUG] [app] Starting request: urn:request:0b7df422-a9dd-46f3-8d31-e89289a71af5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:03:00,640 [284] [DEBUG] [app] Ending request: urn:request:0b7df422-a9dd-46f3-8d31-e89289a71af5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b7df422-a9dd-46f3-8d31-e89289a71af5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:00,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:03:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:00,660 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:00,660 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:00,673 [286] [DEBUG] [app] Ending request: urn:request:e4290c3f-dc06-4e4a-b6e3-fdaf162cfe66 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4290c3f-dc06-4e4a-b6e3-fdaf162cfe66', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:00,673 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:00,673 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.041 118 0.041)
gunicorn-web stdout | 2024-04-22 04:03:00,674 [282] [DEBUG] [app] Ending request: urn:request:9f8c805c-e462-43ed-93ac-fefae240ae15 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9f8c805c-e462-43ed-93ac-fefae240ae15', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:00,674 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:00,675 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.043 118 0.043)
gcworker stdout | 2024-04-22 04:03:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:03:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:31.842666+00:00 (in 29.999482 seconds)
gcworker stdout | 2024-04-22 04:03:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:31 UTC)" (scheduled at 2024-04-22 04:03:01.842666+00:00)
gcworker stdout | 2024-04-22 04:03:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712548981843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:03:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:03:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:03:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:03:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:03:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:09.033206+00:00 (in 6.997057 seconds)
repositorygcworker stdout | 2024-04-22 04:03:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:02 UTC)" (scheduled at 2024-04-22 04:03:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:03:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:03:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:03:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:12.430973+00:00 (in 9.999545 seconds)
notificationworker stdout | 2024-04-22 04:03:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:12 UTC)" (scheduled at 2024-04-22 04:03:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:03:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:03:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 2, 431843), True, datetime.datetime(2024, 4, 22, 4, 3, 2, 431843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:03:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:03:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:03:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:03:05,036 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:03:06,332 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:03:08,942 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 04:03:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:03:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:26.035364+00:00 (in 17.001676 seconds)
repositorygcworker stdout | 2024-04-22 04:03:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:09 UTC)" (scheduled at 2024-04-22 04:03:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:03:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:03:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 9, 34138), True, datetime.datetime(2024, 4, 22, 4, 3, 9, 34138), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:03:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:03:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:03:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:03:09,985 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:03:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:03:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:40.238173+00:00 (in 29.999465 seconds)
autopruneworker stdout | 2024-04-22 04:03:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:40 UTC)" (scheduled at 2024-04-22 04:03:10.238173+00:00)
namespacegcworker stdout | 2024-04-22 04:03:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:03:10,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758530248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:03:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:47.245913+00:00 (in 36.996843 seconds)
namespacegcworker stdout | 2024-04-22 04:03:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:10 UTC)" (scheduled at 2024-04-22 04:03:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:03:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:03:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:03:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:03:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:03:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:03:10,355 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:03:10,395 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:03:11,012 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:03:11,671 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:03:11,893 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:03:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:22.430973+00:00 (in 9.999461 seconds)
notificationworker stdout | 2024-04-22 04:03:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:22 UTC)" (scheduled at 2024-04-22 04:03:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:03:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:03:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 12, 431921), True, datetime.datetime(2024, 4, 22, 4, 3, 12, 431921), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:03:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:03:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:03:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:22 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:03:13,204 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:03:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:03:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:17.065745+00:00 (in 2.998163 seconds)
securityworker stdout | 2024-04-22 04:03:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:44 UTC)" (scheduled at 2024-04-22 04:03:14.067050+00:00)
securityworker stdout | 2024-04-22 04:03:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:03:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:03:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:03:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:03:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:03:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:03:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:03:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:03:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:03:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:03:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 58, 14, 77961), 1, 2])
securityworker stdout | 2024-04-22 04:03:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:03:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:03:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 58, 14, 77961), 1, 2])
securityworker stdout | 2024-04-22 04:03:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:03:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:03:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:03:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:03:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:03:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:03:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:03:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:03:14,132 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:03:15,632 [284] [DEBUG] [app] Starting request: urn:request:0a6c85ed-9532-4d5b-9720-d45419817276 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:15,632 [286] [DEBUG] [app] Starting request: urn:request:206e09ec-349a-4c62-a035-909f8b743baf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:03:15,635 [275] [DEBUG] [app] Starting request: urn:request:65475261-27c4-4248-8f38-f52d8676a817 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:15,635 [278] [DEBUG] [app] Starting request: urn:request:69a406c6-4da4-44d5-a77a-e73ecce96c9e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:15,636 [275] [DEBUG] [app] Ending request: urn:request:65475261-27c4-4248-8f38-f52d8676a817 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:65475261-27c4-4248-8f38-f52d8676a817', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:03:15,636 [278] [DEBUG] [app] Ending request: urn:request:69a406c6-4da4-44d5-a77a-e73ecce96c9e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:69a406c6-4da4-44d5-a77a-e73ecce96c9e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:03:15,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:03:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:03:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:15,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:15,639 [286] [DEBUG] [app] Starting request: urn:request:81299e4b-4abf-4368-a6ef-c060cb82a4a1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:03:15,639 [286] [DEBUG] [app] Ending request: urn:request:81299e4b-4abf-4368-a6ef-c060cb82a4a1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:81299e4b-4abf-4368-a6ef-c060cb82a4a1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:15,639 [285] [DEBUG] [app] Starting request: urn:request:2d1564a6-c43f-4a83-b347-2f1f7410563a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:03:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:03:15,639 [285] [DEBUG] [app] Ending request: urn:request:2d1564a6-c43f-4a83-b347-2f1f7410563a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2d1564a6-c43f-4a83-b347-2f1f7410563a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:03:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:15,660 [286] [DEBUG] [app] Ending request: urn:request:206e09ec-349a-4c62-a035-909f8b743baf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:206e09ec-349a-4c62-a035-909f8b743baf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:15,660 [284] [DEBUG] [app] Ending request: urn:request:0a6c85ed-9532-4d5b-9720-d45419817276 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0a6c85ed-9532-4d5b-9720-d45419817276', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:03:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:03:15,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:03:16,592 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:03:16,828 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:03:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:03:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:44.067050+00:00 (in 27.000742 seconds)
securityworker stdout | 2024-04-22 04:03:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:47 UTC)" (scheduled at 2024-04-22 04:03:17.065745+00:00)
securityworker stdout | 2024-04-22 04:03:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:03:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:03:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:03:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:03:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:03:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:03:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:03:17,277 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:03:17,947 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 04:03:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:03:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:53.045554+00:00 (in 34.997873 seconds)
exportactionlogsworker stdout | 2024-04-22 04:03:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:18 UTC)" (scheduled at 2024-04-22 04:03:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:03:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:03:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:03:18,913 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:03:19,797 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:03:19,799 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:03:19,802 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:03:19,906 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:03:20,385 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:03:21,302 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:03:21,364 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:03:21,371 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:03:21,376 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:03:21,384 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:03:21,388 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:03:21,401 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:03:21,403 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:03:21,407 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:03:21,478 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:03:21,486 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:03:21,489 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:03:21,493 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:03:21,495 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:03:21,586 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:03:21,902 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:03:22,081 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:03:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:32.430973+00:00 (in 9.999499 seconds)
notificationworker stdout | 2024-04-22 04:03:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:32 UTC)" (scheduled at 2024-04-22 04:03:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:03:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:03:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 22, 431911), True, datetime.datetime(2024, 4, 22, 4, 3, 22, 431911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:03:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:03:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:03:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:32 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:03:22,874 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:03:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:03:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:24.738365+00:00 (in 59.999469 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:03:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:24 UTC)" (scheduled at 2024-04-22 04:03:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:03:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:03:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:03:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:03:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:03:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:02.035613+00:00 (in 35.999745 seconds)
repositorygcworker stdout | 2024-04-22 04:03:26,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:08:26 UTC)" (scheduled at 2024-04-22 04:03:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 04:03:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 3, 26, 36278), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:03:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 3, 26, 36278), True, datetime.datetime(2024, 4, 22, 4, 3, 26, 36278), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:03:26,055 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 3, 26, 36278), True, datetime.datetime(2024, 4, 22, 4, 3, 26, 36278), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 4, 3, 26, 36278), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:03:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:03:26,059 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:08:26 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:03:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:03:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:58.130973+00:00 (in 29.999471 seconds)
buildlogsarchiver stdout | 2024-04-22 04:03:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:58 UTC)" (scheduled at 2024-04-22 04:03:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:03:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 3, 28, 131922), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:03:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:03:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:03:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:03:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:30.933445+00:00 (in 1.001974 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:29 UTC)" (scheduled at 2024-04-22 04:03:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 29, 931902), True, datetime.datetime(2024, 4, 22, 4, 3, 29, 931902), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:03:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:03:30,632 [282] [DEBUG] [app] Starting request: urn:request:5e7fc13e-80e9-484d-b79d-83a06de493ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:30,632 [286] [DEBUG] [app] Starting request: urn:request:967836d4-3b42-4c7e-adbc-a4a2522b547d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:03:30,635 [283] [DEBUG] [app] Starting request: urn:request:7c036ff4-b665-45cf-9fbb-cc0f5a1178a8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:30,636 [283] [DEBUG] [app] Ending request: urn:request:7c036ff4-b665-45cf-9fbb-cc0f5a1178a8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7c036ff4-b665-45cf-9fbb-cc0f5a1178a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:03:30,636 [275] [DEBUG] [app] Starting request: urn:request:2c1e3f99-b407-476c-9008-a8542b7146a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:03:30,636 [275] [DEBUG] [app] Ending request: urn:request:2c1e3f99-b407-476c-9008-a8542b7146a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c1e3f99-b407-476c-9008-a8542b7146a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:03:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:30,640 [285] [DEBUG] [app] Starting request: urn:request:9f4cbaa4-b78e-46e0-977b-9ac87acca98d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:03:30,640 [285] [DEBUG] [app] Ending request: urn:request:9f4cbaa4-b78e-46e0-977b-9ac87acca98d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9f4cbaa4-b78e-46e0-977b-9ac87acca98d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:30,640 [284] [DEBUG] [app] Starting request: urn:request:59d9f5a9-104c-4bf1-9b77-6cec71a990f4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:03:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:30,640 [284] [DEBUG] [app] Ending request: urn:request:59d9f5a9-104c-4bf1-9b77-6cec71a990f4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:59d9f5a9-104c-4bf1-9b77-6cec71a990f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:03:30,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:30,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:30,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:30,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:30,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:30,662 [286] [DEBUG] [app] Ending request: urn:request:967836d4-3b42-4c7e-adbc-a4a2522b547d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:967836d4-3b42-4c7e-adbc-a4a2522b547d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:30,662 [282] [DEBUG] [app] Ending request: urn:request:5e7fc13e-80e9-484d-b79d-83a06de493ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5e7fc13e-80e9-484d-b79d-83a06de493ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:30,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:30,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:30,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:03:30,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:31.933721+00:00 (in 0.999757 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:08:30 UTC)" (scheduled at 2024-04-22 04:03:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 3, 30, 934222), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,947 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 3, 30, 934222), True, datetime.datetime(2024, 4, 22, 4, 3, 30, 934222), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,951 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 3, 30, 934222), True, datetime.datetime(2024, 4, 22, 4, 3, 30, 934222), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 4, 3, 30, 934222), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,955 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:03:30,955 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:08:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:03:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:03:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:01.842666+00:00 (in 29.999491 seconds)
gcworker stdout | 2024-04-22 04:03:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:01 UTC)" (scheduled at 2024-04-22 04:03:31.842666+00:00)
gcworker stdout | 2024-04-22 04:03:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549011843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:03:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:03:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:03:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:03:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:03:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:29.931002+00:00 (in 57.996813 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:03:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:31 UTC)" (scheduled at 2024-04-22 04:03:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:03:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:03:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:03:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:42.430973+00:00 (in 9.999573 seconds)
notificationworker stdout | 2024-04-22 04:03:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:42 UTC)" (scheduled at 2024-04-22 04:03:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:03:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:03:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 32, 431789), True, datetime.datetime(2024, 4, 22, 4, 3, 32, 431789), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:03:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:03:32,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:03:32,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:03:35,072 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:03:36,369 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:03:38,979 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:03:40,022 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:03:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:03:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:10.238173+00:00 (in 29.999557 seconds)
autopruneworker stdout | 2024-04-22 04:03:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:10 UTC)" (scheduled at 2024-04-22 04:03:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:03:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758560247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:03:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:03:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:03:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:03:40,368 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:03:40,432 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:03:41,049 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:03:41,685 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:03:41,900 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:03:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:52.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 04:03:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:52 UTC)" (scheduled at 2024-04-22 04:03:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:03:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:03:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 42, 431934), True, datetime.datetime(2024, 4, 22, 4, 3, 42, 431934), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:03:42,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:03:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:03:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:03:52 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:03:43,234 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:03:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:03:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:47.065745+00:00 (in 2.998138 seconds)
securityworker stdout | 2024-04-22 04:03:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:14 UTC)" (scheduled at 2024-04-22 04:03:44.067050+00:00)
securityworker stdout | 2024-04-22 04:03:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:03:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:03:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:03:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:03:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:03:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:03:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:03:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:03:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:03:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:03:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 58, 44, 77737), 1, 2])
securityworker stdout | 2024-04-22 04:03:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:03:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:03:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 58, 44, 77737), 1, 2])
securityworker stdout | 2024-04-22 04:03:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:03:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:03:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:03:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:03:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:03:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:03:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:03:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:03:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:03:44,170 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:03:45,632 [282] [DEBUG] [app] Starting request: urn:request:74f40f75-84d3-4a97-9a41-7169180649b8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:45,632 [284] [DEBUG] [app] Starting request: urn:request:9d8fcfee-2f49-461b-a649-f8254cd1aaf1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:03:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:03:45,635 [275] [DEBUG] [app] Starting request: urn:request:c40e30af-5891-4b2c-8f10-5ade303f8c0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:45,635 [276] [DEBUG] [app] Starting request: urn:request:78dcaf01-38c6-4854-83b1-504ab8237368 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:03:45,635 [275] [DEBUG] [app] Ending request: urn:request:c40e30af-5891-4b2c-8f10-5ade303f8c0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c40e30af-5891-4b2c-8f10-5ade303f8c0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:03:45,635 [276] [DEBUG] [app] Ending request: urn:request:78dcaf01-38c6-4854-83b1-504ab8237368 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78dcaf01-38c6-4854-83b1-504ab8237368', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:03:45,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:03:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:03:45,638 [282] [DEBUG] [app] Starting request: urn:request:25ad40e1-784f-4b3f-a84c-82748f8b2be8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:03:45,639 [282] [DEBUG] [app] Ending request: urn:request:25ad40e1-784f-4b3f-a84c-82748f8b2be8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25ad40e1-784f-4b3f-a84c-82748f8b2be8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:03:45,639 [285] [DEBUG] [app] Starting request: urn:request:3ff7ae89-f094-4abc-84ac-0e5a3b3f7f3b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:03:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:45,639 [285] [DEBUG] [app] Ending request: urn:request:3ff7ae89-f094-4abc-84ac-0e5a3b3f7f3b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3ff7ae89-f094-4abc-84ac-0e5a3b3f7f3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:03:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:03:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:03:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:03:45,639 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:03:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:03:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:03:45,660 [284] [DEBUG] [app] Ending request: urn:request:9d8fcfee-2f49-461b-a649-f8254cd1aaf1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d8fcfee-2f49-461b-a649-f8254cd1aaf1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:45,660 [282] [DEBUG] [app] Ending request: urn:request:74f40f75-84d3-4a97-9a41-7169180649b8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:74f40f75-84d3-4a97-9a41-7169180649b8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:03:45,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:03:45,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:03:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:03:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:03:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 04:03:46,628 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:03:46,863 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:03:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:03:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:14.067050+00:00 (in 27.000740 seconds)
securityworker stdout | 2024-04-22 04:03:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:17 UTC)" (scheduled at 2024-04-22 04:03:47.065745+00:00)
securityworker stdout | 2024-04-22 04:03:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:03:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:03:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:03:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:03:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:03:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:03:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:03:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:03:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:10.248668+00:00 (in 23.002270 seconds)
namespacegcworker stdout | 2024-04-22 04:03:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:47 UTC)" (scheduled at 2024-04-22 04:03:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:03:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:03:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 47, 246656), True, datetime.datetime(2024, 4, 22, 4, 3, 47, 246656), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:03:47,265 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:03:47,265 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:03:47,265 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:03:47,312 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:03:47,984 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:03:48,944 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:03:49,806 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:03:49,808 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:03:49,810 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:03:49,943 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:03:50,422 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:03:51,313 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:03:51,374 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:03:51,380 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:03:51,385 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:03:51,394 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:03:51,398 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:03:51,410 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:03:51,413 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:03:51,415 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:03:51,485 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:03:51,493 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:03:51,495 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:03:51,500 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:03:51,502 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:03:51,622 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
quotatotalworker stdout | 2024-04-22 04:03:51,804 [86] [DEBUG] [workers.worker] Scheduling worker.
quotatotalworker stdout | 2024-04-22 04:03:51,805 [86] [INFO] [apscheduler.scheduler] Scheduler started
quotatotalworker stdout | 2024-04-22 04:03:51,805 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:03:51,805 [86] [DEBUG] [apscheduler.scheduler] No jobs; waiting until a job is added
quotatotalworker stdout | 2024-04-22 04:03:51,805 [86] [DEBUG] [workers.worker] First run scheduled for 2024-04-22 04:03:57.806778
quotatotalworker stdout | 2024-04-22 04:03:51,807 [86] [INFO] [apscheduler.scheduler] Added job "QuotaTotalWorker.backfill" to job store "default"
quotatotalworker stdout | 2024-04-22 04:03:51,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:03:51,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:57.806778+00:00 (in 5.999372 seconds)
namespacegcworker stdout | 2024-04-22 04:03:51,938 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:03:52,117 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:03:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:03:54.433683+00:00 (in 2.002165 seconds)
notificationworker stdout | 2024-04-22 04:03:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:02 UTC)" (scheduled at 2024-04-22 04:03:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:03:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:03:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 52, 431958), True, datetime.datetime(2024, 4, 22, 4, 3, 52, 431958), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:03:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:03:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:03:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:02 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:03:52,868 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:03:52,910 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:03:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:03:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:18.047111+00:00 (in 25.001043 seconds)
exportactionlogsworker stdout | 2024-04-22 04:03:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:53 UTC)" (scheduled at 2024-04-22 04:03:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:03:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:03:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 3, 53, 46350), True, datetime.datetime(2024, 4, 22, 4, 3, 53, 46350), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:03:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:03:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:03:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:03:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:03:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:02.430973+00:00 (in 7.996786 seconds)
notificationworker stdout | 2024-04-22 04:03:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:54 UTC)" (scheduled at 2024-04-22 04:03:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:03:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:03:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:04:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:03:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:03:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:03:57 UTC)" (scheduled at 2024-04-22 04:03:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:03:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:03:57,809 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:12.806778+00:00 (in 14.997353 seconds)
quotatotalworker stdout | 2024-04-22 04:03:57,823 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
quotatotalworker stdout | 2024-04-22 04:03:57,827 [86] [DEBUG] [peewee] ('INSERT INTO "quotanamespacesize" ("namespace_user_id", "size_bytes", "backfill_start_ms", "backfill_complete") VALUES (%s, %s, %s, %s) RETURNING "quotanamespacesize"."id"', [1, 0, 1713758637827, False])
quotatotalworker stdout | 2024-04-22 04:03:57,833 [86] [DEBUG] [peewee] ('SELECT Sum("t1"."image_size") FROM (SELECT "t2"."image_size" FROM "imagestorage" AS "t2" INNER JOIN "manifestblob" AS "t3" ON ("t2"."id" = "t3"."blob_id") INNER JOIN "repository" AS "t4" ON ("t4"."id" = "t3"."repository_id") WHERE ("t4"."namespace_user_id" = %s) GROUP BY "t2"."id") AS "t1"', [1])
quotatotalworker stdout | 2024-04-22 04:03:57,840 [86] [DEBUG] [peewee] ('UPDATE "quotanamespacesize" SET "size_bytes" = %s, "backfill_complete" = %s WHERE ("quotanamespacesize"."namespace_user_id" = %s)', [0, True, 1])
quotatotalworker stdout | 2024-04-22 04:03:57,846 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE (("t1"."namespace_user_id" = %s) AND ("t1"."state" != %s))', [1, 3])
quotatotalworker stdout | 2024-04-22 04:03:57,850 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
quotatotalworker stdout | 2024-04-22 04:03:57,853 [86] [DEBUG] [peewee] ('INSERT INTO "quotanamespacesize" ("namespace_user_id", "size_bytes", "backfill_start_ms", "backfill_complete") VALUES (%s, %s, %s, %s) RETURNING "quotanamespacesize"."id"', [2, 0, 1713758637853, False])
quotatotalworker stdout | 2024-04-22 04:03:57,859 [86] [DEBUG] [peewee] ('SELECT Sum("t1"."image_size") FROM (SELECT "t2"."image_size" FROM "imagestorage" AS "t2" INNER JOIN "manifestblob" AS "t3" ON ("t2"."id" = "t3"."blob_id") INNER JOIN "repository" AS "t4" ON ("t4"."id" = "t3"."repository_id") WHERE ("t4"."namespace_user_id" = %s) GROUP BY "t2"."id") AS "t1"', [2])
quotatotalworker stdout | 2024-04-22 04:03:57,863 [86] [DEBUG] [peewee] ('UPDATE "quotanamespacesize" SET "size_bytes" = %s, "backfill_complete" = %s WHERE ("quotanamespacesize"."namespace_user_id" = %s)', [0, True, 2])
quotatotalworker stdout | 2024-04-22 04:03:57,868 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE (("t1"."namespace_user_id" = %s) AND ("t1"."state" != %s))', [2, 3])
quotatotalworker stdout | 2024-04-22 04:03:57,872 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:03:57,872 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:03:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:03:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:28.130973+00:00 (in 29.999464 seconds)
buildlogsarchiver stdout | 2024-04-22 04:03:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:28 UTC)" (scheduled at 2024-04-22 04:03:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:03:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 3, 58, 131909), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:03:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:03:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:03:58,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:04:00,632 [286] [DEBUG] [app] Starting request: urn:request:fc08e3ac-03b4-4234-bbfe-8d65c1c6e8d4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:00,632 [285] [DEBUG] [app] Starting request: urn:request:70332714-67c5-4e3b-ac75-76096c8d0283 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:04:00,635 [279] [DEBUG] [app] Starting request: urn:request:cb77dd59-77f2-4d49-a0ce-a9ec4d9eb30a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:00,635 [283] [DEBUG] [app] Starting request: urn:request:ccc032e1-717c-4d64-b9ca-f955ef76fc2e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:00,636 [283] [DEBUG] [app] Ending request: urn:request:ccc032e1-717c-4d64-b9ca-f955ef76fc2e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ccc032e1-717c-4d64-b9ca-f955ef76fc2e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:00,636 [279] [DEBUG] [app] Ending request: urn:request:cb77dd59-77f2-4d49-a0ce-a9ec4d9eb30a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb77dd59-77f2-4d49-a0ce-a9ec4d9eb30a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:04:00,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:04:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:00,639 [282] [DEBUG] [app] Starting request: urn:request:998720b6-18cf-48f6-9e39-c24e90b387d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:00,639 [282] [DEBUG] [app] Ending request: urn:request:998720b6-18cf-48f6-9e39-c24e90b387d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:998720b6-18cf-48f6-9e39-c24e90b387d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:00,639 [284] [DEBUG] [app] Starting request: urn:request:65f35280-1fdc-4040-8009-c4b9a56be601 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:04:00,640 [284] [DEBUG] [app] Ending request: urn:request:65f35280-1fdc-4040-8009-c4b9a56be601 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:65f35280-1fdc-4040-8009-c4b9a56be601', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:04:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:04:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:00,661 [286] [DEBUG] [app] Ending request: urn:request:fc08e3ac-03b4-4234-bbfe-8d65c1c6e8d4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fc08e3ac-03b4-4234-bbfe-8d65c1c6e8d4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:00,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:04:00,667 [285] [DEBUG] [app] Ending request: urn:request:70332714-67c5-4e3b-ac75-76096c8d0283 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:70332714-67c5-4e3b-ac75-76096c8d0283', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:00,667 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:00,667 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.036)
gcworker stdout | 2024-04-22 04:04:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:04:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:31.842666+00:00 (in 29.999469 seconds)
gcworker stdout | 2024-04-22 04:04:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:31 UTC)" (scheduled at 2024-04-22 04:04:01.842666+00:00)
gcworker stdout | 2024-04-22 04:04:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549041843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:04:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:04:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:04:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:04:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:04:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:09.033206+00:00 (in 6.997039 seconds)
repositorygcworker stdout | 2024-04-22 04:04:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:02 UTC)" (scheduled at 2024-04-22 04:04:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:04:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:04:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:04:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:12.430973+00:00 (in 9.999496 seconds)
notificationworker stdout | 2024-04-22 04:04:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:12 UTC)" (scheduled at 2024-04-22 04:04:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:04:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:04:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 2, 431875), True, datetime.datetime(2024, 4, 22, 4, 4, 2, 431875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:04:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:04:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:04:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:04:05,109 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:04:06,401 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:04:09,015 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
repositorygcworker stdout | 2024-04-22 04:04:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:04:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:02.035613+00:00 (in 53.001939 seconds)
repositorygcworker stdout | 2024-04-22 04:04:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:09 UTC)" (scheduled at 2024-04-22 04:04:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:04:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:04:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 9, 33957), True, datetime.datetime(2024, 4, 22, 4, 4, 9, 33957), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:04:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:04:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:04:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:09 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:04:10,041 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:04:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:04:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:40.238173+00:00 (in 29.999537 seconds)
autopruneworker stdout | 2024-04-22 04:04:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:40 UTC)" (scheduled at 2024-04-22 04:04:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:04:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758590247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:04:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:04:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:47.245913+00:00 (in 36.996794 seconds)
namespacegcworker stdout | 2024-04-22 04:04:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:10 UTC)" (scheduled at 2024-04-22 04:04:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:04:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:04:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:04:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:04:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:04:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:04:10,400 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:04:10,468 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:04:11,057 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:04:11,721 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:04:11,934 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:04:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:22.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 04:04:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:22 UTC)" (scheduled at 2024-04-22 04:04:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:04:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:04:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 12, 431804), True, datetime.datetime(2024, 4, 22, 4, 4, 12, 431804), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:04:12,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:04:12,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:04:12,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:04:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:04:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:27.806778+00:00 (in 14.999380 seconds)
quotatotalworker stdout | 2024-04-22 04:04:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:27 UTC)" (scheduled at 2024-04-22 04:04:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:04:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:04:12,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:04:12,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:04:13,266 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:04:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:04:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:17.065745+00:00 (in 2.998162 seconds)
securityworker stdout | 2024-04-22 04:04:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:44 UTC)" (scheduled at 2024-04-22 04:04:14.067050+00:00)
securityworker stdout | 2024-04-22 04:04:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:04:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:04:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:04:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:04:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:04:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:04:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:04:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:04:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:04:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:04:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 59, 14, 77633), 1, 2])
securityworker stdout | 2024-04-22 04:04:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:04:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:04:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 59, 14, 77633), 1, 2])
securityworker stdout | 2024-04-22 04:04:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:04:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:04:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:04:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:04:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:04:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:04:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:04:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:04:14,184 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:04:15,632 [282] [DEBUG] [app] Starting request: urn:request:b5c4ea0f-17e8-40d9-87d1-423976ca5850 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:15,632 [286] [DEBUG] [app] Starting request: urn:request:1a56dce5-5d57-42af-88b2-426a0ca025e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:04:15,636 [277] [DEBUG] [app] Starting request: urn:request:d8969303-8c86-43ff-8f27-2f7fd672e2c1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:15,636 [281] [DEBUG] [app] Starting request: urn:request:f4d90dd1-808e-41d9-b144-04fe06a5f792 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:15,636 [277] [DEBUG] [app] Ending request: urn:request:d8969303-8c86-43ff-8f27-2f7fd672e2c1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d8969303-8c86-43ff-8f27-2f7fd672e2c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:15,636 [281] [DEBUG] [app] Ending request: urn:request:f4d90dd1-808e-41d9-b144-04fe06a5f792 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4d90dd1-808e-41d9-b144-04fe06a5f792', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:04:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:04:15,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:15,639 [286] [DEBUG] [app] Starting request: urn:request:26e79c22-0e2b-42c9-a610-ba65283f3577 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:15,639 [286] [DEBUG] [app] Ending request: urn:request:26e79c22-0e2b-42c9-a610-ba65283f3577 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26e79c22-0e2b-42c9-a610-ba65283f3577', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:15,639 [285] [DEBUG] [app] Starting request: urn:request:3d45ffd6-516f-4e3e-9623-51b6f54ae076 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:04:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:15,639 [285] [DEBUG] [app] Ending request: urn:request:3d45ffd6-516f-4e3e-9623-51b6f54ae076 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3d45ffd6-516f-4e3e-9623-51b6f54ae076', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:04:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:04:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:15,660 [282] [DEBUG] [app] Ending request: urn:request:b5c4ea0f-17e8-40d9-87d1-423976ca5850 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5c4ea0f-17e8-40d9-87d1-423976ca5850', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:15,661 [286] [DEBUG] [app] Ending request: urn:request:1a56dce5-5d57-42af-88b2-426a0ca025e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1a56dce5-5d57-42af-88b2-426a0ca025e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:15,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:04:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:04:16,652 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:04:16,875 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:04:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:04:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:44.067050+00:00 (in 27.000759 seconds)
securityworker stdout | 2024-04-22 04:04:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:47 UTC)" (scheduled at 2024-04-22 04:04:17.065745+00:00)
securityworker stdout | 2024-04-22 04:04:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:04:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:04:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:04:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:04:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:04:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:04:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:04:17,341 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:04:18,001 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 04:04:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:04:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:53.045554+00:00 (in 34.997933 seconds)
exportactionlogsworker stdout | 2024-04-22 04:04:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:18 UTC)" (scheduled at 2024-04-22 04:04:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:04:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:04:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:04:18,976 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:04:19,814 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:04:19,816 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:04:19,820 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:04:19,975 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:04:20,455 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:04:21,323 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:04:21,384 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:04:21,389 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:04:21,392 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:04:21,405 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:04:21,408 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:04:21,418 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:04:21,422 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 04:04:21,424 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 04:04:21,493 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:04:21,501 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:04:21,504 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:04:21,506 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:04:21,509 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:04:21,658 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:04:21,964 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:04:22,132 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:04:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:32.430973+00:00 (in 9.999522 seconds)
notificationworker stdout | 2024-04-22 04:04:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:32 UTC)" (scheduled at 2024-04-22 04:04:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:04:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:04:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 22, 431712), True, datetime.datetime(2024, 4, 22, 4, 4, 22, 431712), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:04:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:04:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:04:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:04:22,905 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:04:22,945 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:04:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:04:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:24.738365+00:00 (in 59.999463 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:04:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:24 UTC)" (scheduled at 2024-04-22 04:04:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:04:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:04:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:04:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:04:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:04:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:42.806778+00:00 (in 14.999503 seconds)
quotatotalworker stdout | 2024-04-22 04:04:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:42 UTC)" (scheduled at 2024-04-22 04:04:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:04:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:04:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:04:27,823 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:04:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:04:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:58.130973+00:00 (in 29.999484 seconds)
buildlogsarchiver stdout | 2024-04-22 04:04:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:58 UTC)" (scheduled at 2024-04-22 04:04:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:04:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 4, 28, 131861), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:04:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:04:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:04:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:04:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:31.933721+00:00 (in 2.002203 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:29 UTC)" (scheduled at 2024-04-22 04:04:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 29, 931935), True, datetime.datetime(2024, 4, 22, 4, 4, 29, 931935), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:04:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:04:30,632 [285] [DEBUG] [app] Starting request: urn:request:55b34452-8121-442b-a069-7c7cde4c2826 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:30,632 [284] [DEBUG] [app] Starting request: urn:request:76f70928-9fc8-452a-8555-a3982711cb50 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:30,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:04:30,635 [276] [DEBUG] [app] Starting request: urn:request:48bccd3d-dff9-42f0-965f-93d2cb94ffd0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:30,635 [275] [DEBUG] [app] Starting request: urn:request:f60dd495-e39b-4e21-b70f-99bae51a1057 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:30,635 [275] [DEBUG] [app] Ending request: urn:request:f60dd495-e39b-4e21-b70f-99bae51a1057 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f60dd495-e39b-4e21-b70f-99bae51a1057', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:30,635 [276] [DEBUG] [app] Ending request: urn:request:48bccd3d-dff9-42f0-965f-93d2cb94ffd0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48bccd3d-dff9-42f0-965f-93d2cb94ffd0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:30,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:04:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:04:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:30,638 [285] [DEBUG] [app] Starting request: urn:request:6bbd88bd-6b34-421a-98a6-7f2ee006db33 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:30,639 [285] [DEBUG] [app] Ending request: urn:request:6bbd88bd-6b34-421a-98a6-7f2ee006db33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6bbd88bd-6b34-421a-98a6-7f2ee006db33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:04:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:30,639 [282] [DEBUG] [app] Starting request: urn:request:e14fcdc6-356e-4ae1-9c5b-87849fd0d8e9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:30,640 [282] [DEBUG] [app] Ending request: urn:request:e14fcdc6-356e-4ae1-9c5b-87849fd0d8e9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e14fcdc6-356e-4ae1-9c5b-87849fd0d8e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:30,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:04:30,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:30,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:04:30,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:30,652 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:30,661 [285] [DEBUG] [app] Ending request: urn:request:55b34452-8121-442b-a069-7c7cde4c2826 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:55b34452-8121-442b-a069-7c7cde4c2826', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:30,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:30,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:04:30,664 [284] [DEBUG] [app] Ending request: urn:request:76f70928-9fc8-452a-8555-a3982711cb50 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76f70928-9fc8-452a-8555-a3982711cb50', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:30,664 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:30,665 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.034)
gcworker stdout | 2024-04-22 04:04:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:04:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:01.842666+00:00 (in 29.999482 seconds)
gcworker stdout | 2024-04-22 04:04:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:01 UTC)" (scheduled at 2024-04-22 04:04:31.842666+00:00)
gcworker stdout | 2024-04-22 04:04:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549071843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:04:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:04:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:04:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:04:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:04:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:29.931002+00:00 (in 57.996778 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:04:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:31 UTC)" (scheduled at 2024-04-22 04:04:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:04:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:04:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:04:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:42.430973+00:00 (in 9.999457 seconds)
notificationworker stdout | 2024-04-22 04:04:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:42 UTC)" (scheduled at 2024-04-22 04:04:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:04:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:04:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 32, 431837), True, datetime.datetime(2024, 4, 22, 4, 4, 32, 431837), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:04:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:04:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:04:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:04:35,145 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:04:36,430 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:04:39,048 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:04:40,075 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:04:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:04:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:10.238173+00:00 (in 29.999605 seconds)
autopruneworker stdout | 2024-04-22 04:04:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:10 UTC)" (scheduled at 2024-04-22 04:04:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:04:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758620248, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:04:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:04:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:04:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:04:40,413 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:04:40,476 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:04:41,094 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:04:41,757 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:04:41,968 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:04:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:52.430973+00:00 (in 9.999406 seconds)
notificationworker stdout | 2024-04-22 04:04:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:52 UTC)" (scheduled at 2024-04-22 04:04:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:04:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:04:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 42, 431982), True, datetime.datetime(2024, 4, 22, 4, 4, 42, 431982), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:04:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:04:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:04:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:04:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:04:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:04:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:57.806778+00:00 (in 14.999456 seconds)
quotatotalworker stdout | 2024-04-22 04:04:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:57 UTC)" (scheduled at 2024-04-22 04:04:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:04:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:04:42,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:04:42,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:04:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:04:43,277 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:04:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:04:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:47.065745+00:00 (in 2.998191 seconds)
securityworker stdout | 2024-04-22 04:04:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:14 UTC)" (scheduled at 2024-04-22 04:04:44.067050+00:00)
securityworker stdout | 2024-04-22 04:04:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:04:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:04:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:04:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:04:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:04:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:04:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:04:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:04:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:04:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:04:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 3, 59, 44, 77730), 1, 2])
securityworker stdout | 2024-04-22 04:04:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:04:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:04:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 3, 59, 44, 77730), 1, 2])
securityworker stdout | 2024-04-22 04:04:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:04:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:04:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:04:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:04:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:04:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:04:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:04:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:04:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:04:44,195 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:04:45,632 [282] [DEBUG] [app] Starting request: urn:request:146e0182-7035-4357-8409-21bac330c3d0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:45,632 [285] [DEBUG] [app] Starting request: urn:request:6ecc8aa4-8866-40c2-883c-955ac721127c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:04:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:04:45,635 [275] [DEBUG] [app] Starting request: urn:request:aa469ef7-5076-4631-9a06-92bcd306faa0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:45,635 [283] [DEBUG] [app] Starting request: urn:request:c038c54e-f56f-424e-ad20-27d4f8699ff5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:04:45,636 [275] [DEBUG] [app] Ending request: urn:request:aa469ef7-5076-4631-9a06-92bcd306faa0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aa469ef7-5076-4631-9a06-92bcd306faa0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:45,636 [283] [DEBUG] [app] Ending request: urn:request:c038c54e-f56f-424e-ad20-27d4f8699ff5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c038c54e-f56f-424e-ad20-27d4f8699ff5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:04:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:04:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:04:45,638 [285] [DEBUG] [app] Starting request: urn:request:b111ecf1-f4f9-4e46-ab9a-5633bb2613cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:45,639 [285] [DEBUG] [app] Ending request: urn:request:b111ecf1-f4f9-4e46-ab9a-5633bb2613cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b111ecf1-f4f9-4e46-ab9a-5633bb2613cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:45,639 [286] [DEBUG] [app] Starting request: urn:request:3b160a84-ec5d-475d-b9dc-295e64d9593b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:04:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:04:45,639 [286] [DEBUG] [app] Ending request: urn:request:3b160a84-ec5d-475d-b9dc-295e64d9593b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3b160a84-ec5d-475d-b9dc-295e64d9593b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:04:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:04:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:04:45,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:04:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:04:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:04:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:04:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:04:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:45,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:04:45,660 [285] [DEBUG] [app] Ending request: urn:request:6ecc8aa4-8866-40c2-883c-955ac721127c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ecc8aa4-8866-40c2-883c-955ac721127c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:45,660 [282] [DEBUG] [app] Ending request: urn:request:146e0182-7035-4357-8409-21bac330c3d0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:146e0182-7035-4357-8409-21bac330c3d0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:04:45,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:04:45,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:04:45,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:04:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:04:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:04:46,685 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:04:46,911 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:04:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:04:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:14.067050+00:00 (in 27.000795 seconds)
securityworker stdout | 2024-04-22 04:04:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:17 UTC)" (scheduled at 2024-04-22 04:04:47.065745+00:00)
securityworker stdout | 2024-04-22 04:04:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:04:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:04:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:04:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:04:47,085 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:04:47,089 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:04:47,089 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:04:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:04:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:10.248668+00:00 (in 23.002242 seconds)
namespacegcworker stdout | 2024-04-22 04:04:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:47 UTC)" (scheduled at 2024-04-22 04:04:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:04:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:04:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 47, 246701), True, datetime.datetime(2024, 4, 22, 4, 4, 47, 246701), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:04:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:04:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:04:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:04:47,373 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:04:48,013 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:04:49,012 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:04:49,823 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:04:49,825 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:04:49,829 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:04:49,996 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:04:50,479 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:04:51,331 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:04:51,393 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:04:51,400 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:04:51,402 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:04:51,414 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:04:51,417 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:04:51,425 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:04:51,431 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 04:04:51,433 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 04:04:51,500 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:04:51,509 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:04:51,511 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:04:51,514 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:04:51,516 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:04:51,695 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:04:52,000 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:04:52,148 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:04:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:04:54.433683+00:00 (in 2.002200 seconds)
notificationworker stdout | 2024-04-22 04:04:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:02 UTC)" (scheduled at 2024-04-22 04:04:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:04:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:04:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 52, 431895), True, datetime.datetime(2024, 4, 22, 4, 4, 52, 431895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:04:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:04:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:04:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:02 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:04:52,941 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:04:52,972 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:04:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:04:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:18.047111+00:00 (in 25.000980 seconds)
exportactionlogsworker stdout | 2024-04-22 04:04:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:53 UTC)" (scheduled at 2024-04-22 04:04:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:04:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:04:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 4, 53, 46554), True, datetime.datetime(2024, 4, 22, 4, 4, 53, 46554), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:04:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:04:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:04:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:04:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:04:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:02.430973+00:00 (in 7.996770 seconds)
notificationworker stdout | 2024-04-22 04:04:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:54 UTC)" (scheduled at 2024-04-22 04:04:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:04:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:04:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:05:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:04:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:04:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:12.806778+00:00 (in 14.999488 seconds)
quotatotalworker stdout | 2024-04-22 04:04:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:12 UTC)" (scheduled at 2024-04-22 04:04:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:04:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:04:57,824 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:04:57,824 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:04:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:04:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:28.130973+00:00 (in 29.999473 seconds)
buildlogsarchiver stdout | 2024-04-22 04:04:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:28 UTC)" (scheduled at 2024-04-22 04:04:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:04:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 4, 58, 131746), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:04:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:04:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:04:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:05:00,632 [282] [DEBUG] [app] Starting request: urn:request:4646c571-1f45-4d2e-b090-4f06451112c9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:00,632 [286] [DEBUG] [app] Starting request: urn:request:315b5426-1d30-4a4b-a5b6-f409064c96bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:05:00,635 [275] [DEBUG] [app] Starting request: urn:request:eb0b4e82-ff6f-43d0-b017-e2edda332e21 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:00,635 [279] [DEBUG] [app] Starting request: urn:request:fb855f81-9556-424a-9296-3f79c5b253ac (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:00,636 [275] [DEBUG] [app] Ending request: urn:request:eb0b4e82-ff6f-43d0-b017-e2edda332e21 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:eb0b4e82-ff6f-43d0-b017-e2edda332e21', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:00,636 [279] [DEBUG] [app] Ending request: urn:request:fb855f81-9556-424a-9296-3f79c5b253ac (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fb855f81-9556-424a-9296-3f79c5b253ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:05:00,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:00,640 [285] [DEBUG] [app] Starting request: urn:request:966f59fb-e2c6-4a26-80a6-0241d92685dd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:00,640 [284] [DEBUG] [app] Starting request: urn:request:ed8aab76-a176-4bff-895d-61048340b84e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:00,640 [285] [DEBUG] [app] Ending request: urn:request:966f59fb-e2c6-4a26-80a6-0241d92685dd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:966f59fb-e2c6-4a26-80a6-0241d92685dd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:00,640 [284] [DEBUG] [app] Ending request: urn:request:ed8aab76-a176-4bff-895d-61048340b84e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ed8aab76-a176-4bff-895d-61048340b84e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:05:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:00,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:00,662 [282] [DEBUG] [app] Ending request: urn:request:4646c571-1f45-4d2e-b090-4f06451112c9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4646c571-1f45-4d2e-b090-4f06451112c9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:00,662 [286] [DEBUG] [app] Ending request: urn:request:315b5426-1d30-4a4b-a5b6-f409064c96bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:315b5426-1d30-4a4b-a5b6-f409064c96bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:00,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:05:00,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gcworker stdout | 2024-04-22 04:05:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:05:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:31.842666+00:00 (in 29.999474 seconds)
gcworker stdout | 2024-04-22 04:05:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:31 UTC)" (scheduled at 2024-04-22 04:05:01.842666+00:00)
gcworker stdout | 2024-04-22 04:05:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549101843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:05:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:05:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:05:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:05:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:05:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:09.033206+00:00 (in 6.997081 seconds)
repositorygcworker stdout | 2024-04-22 04:05:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:02 UTC)" (scheduled at 2024-04-22 04:05:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:05:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:05:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:05:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:12.430973+00:00 (in 9.999487 seconds)
notificationworker stdout | 2024-04-22 04:05:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:12 UTC)" (scheduled at 2024-04-22 04:05:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:05:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:05:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 2, 431761), True, datetime.datetime(2024, 4, 22, 4, 5, 2, 431761), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:05:02,454 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:05:02,454 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:05:02,454 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:05:05,181 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:05:06,466 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:05:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:05:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:02.035613+00:00 (in 53.001919 seconds)
repositorygcworker stdout | 2024-04-22 04:05:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:09 UTC)" (scheduled at 2024-04-22 04:05:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:05:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:05:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 9, 33952), True, datetime.datetime(2024, 4, 22, 4, 5, 9, 33952), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:05:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:05:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:05:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:05:09,074 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:05:10,104 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:05:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:05:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:40.238173+00:00 (in 29.999555 seconds)
autopruneworker stdout | 2024-04-22 04:05:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:40 UTC)" (scheduled at 2024-04-22 04:05:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:05:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758650247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:05:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:05:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:47.245913+00:00 (in 36.996764 seconds)
namespacegcworker stdout | 2024-04-22 04:05:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:10 UTC)" (scheduled at 2024-04-22 04:05:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:05:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:05:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:05:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:05:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:05:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:05:10,419 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:05:10,505 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:05:11,131 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:05:11,767 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:05:11,982 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:05:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:22.430973+00:00 (in 9.999495 seconds)
notificationworker stdout | 2024-04-22 04:05:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:22 UTC)" (scheduled at 2024-04-22 04:05:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:05:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:05:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 12, 431771), True, datetime.datetime(2024, 4, 22, 4, 5, 12, 431771), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:05:12,457 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:05:12,457 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:05:12,457 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:05:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:05:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:27.806778+00:00 (in 14.999472 seconds)
quotatotalworker stdout | 2024-04-22 04:05:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:27 UTC)" (scheduled at 2024-04-22 04:05:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:05:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:05:12,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:05:12,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:05:13,304 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:05:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:05:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:17.065745+00:00 (in 2.998157 seconds)
securityworker stdout | 2024-04-22 04:05:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:44 UTC)" (scheduled at 2024-04-22 04:05:14.067050+00:00)
securityworker stdout | 2024-04-22 04:05:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:05:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:05:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:05:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:05:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:05:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:05:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:05:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:05:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:05:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:05:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 0, 14, 78107), 1, 2])
securityworker stdout | 2024-04-22 04:05:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:05:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:05:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 0, 14, 78107), 1, 2])
securityworker stdout | 2024-04-22 04:05:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:05:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:05:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:05:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:05:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:05:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:05:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:05:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:05:14,233 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:05:15,632 [286] [DEBUG] [app] Starting request: urn:request:a251d8cf-4abe-4e1d-9667-224156b80c5a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:15,632 [282] [DEBUG] [app] Starting request: urn:request:016f187e-2bfc-4e64-9c55-27072e1891fd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:05:15,636 [275] [DEBUG] [app] Starting request: urn:request:477be13b-a2b1-4efd-ada1-53bf8ee2f3e5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:15,636 [280] [DEBUG] [app] Starting request: urn:request:24f91590-dec3-4e83-b2e3-6bd3549bd0d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:15,636 [275] [DEBUG] [app] Ending request: urn:request:477be13b-a2b1-4efd-ada1-53bf8ee2f3e5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:477be13b-a2b1-4efd-ada1-53bf8ee2f3e5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:15,636 [280] [DEBUG] [app] Ending request: urn:request:24f91590-dec3-4e83-b2e3-6bd3549bd0d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24f91590-dec3-4e83-b2e3-6bd3549bd0d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:05:15,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:15,639 [282] [DEBUG] [app] Starting request: urn:request:b9d51081-e7b2-48ee-aba9-d2e16051f09f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:15,639 [282] [DEBUG] [app] Ending request: urn:request:b9d51081-e7b2-48ee-aba9-d2e16051f09f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b9d51081-e7b2-48ee-aba9-d2e16051f09f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:15,639 [284] [DEBUG] [app] Starting request: urn:request:2e5beefd-1e76-4687-956e-7c6c89f3b8b4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:05:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:15,640 [284] [DEBUG] [app] Ending request: urn:request:2e5beefd-1e76-4687-956e-7c6c89f3b8b4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2e5beefd-1e76-4687-956e-7c6c89f3b8b4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:05:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:15,660 [282] [DEBUG] [app] Ending request: urn:request:016f187e-2bfc-4e64-9c55-27072e1891fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:016f187e-2bfc-4e64-9c55-27072e1891fd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:15,660 [286] [DEBUG] [app] Ending request: urn:request:a251d8cf-4abe-4e1d-9667-224156b80c5a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a251d8cf-4abe-4e1d-9667-224156b80c5a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:05:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 04:05:16,721 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:05:16,948 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:05:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:05:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:44.067050+00:00 (in 27.000807 seconds)
securityworker stdout | 2024-04-22 04:05:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:47 UTC)" (scheduled at 2024-04-22 04:05:17.065745+00:00)
securityworker stdout | 2024-04-22 04:05:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:05:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:05:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:05:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:05:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:05:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:05:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:05:17,410 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:05:18,029 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
exportactionlogsworker stdout | 2024-04-22 04:05:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:05:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:53.045554+00:00 (in 34.997910 seconds)
exportactionlogsworker stdout | 2024-04-22 04:05:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:18 UTC)" (scheduled at 2024-04-22 04:05:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:05:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:05:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:18 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:05:19,048 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:05:19,832 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:05:19,835 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:05:19,837 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:05:20,034 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:05:20,502 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:05:21,339 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:05:21,403 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:05:21,409 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:05:21,411 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:05:21,422 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:05:21,424 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:05:21,434 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:05:21,439 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-registry stdout | 2024-04-22 04:05:21,441 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 04:05:21,507 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:05:21,516 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:05:21,519 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:05:21,522 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:05:21,524 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:05:21,731 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:05:22,020 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:05:22,184 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:05:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:32.430973+00:00 (in 9.999465 seconds)
notificationworker stdout | 2024-04-22 04:05:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:32 UTC)" (scheduled at 2024-04-22 04:05:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:05:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:05:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 22, 431796), True, datetime.datetime(2024, 4, 22, 4, 5, 22, 431796), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:05:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:05:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:05:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:05:22,953 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:05:23,001 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:05:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:05:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:24.738365+00:00 (in 59.999475 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:05:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:24 UTC)" (scheduled at 2024-04-22 04:05:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:05:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:05:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:05:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:05:27,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:05:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:42.806778+00:00 (in 14.999527 seconds)
quotatotalworker stdout | 2024-04-22 04:05:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:42 UTC)" (scheduled at 2024-04-22 04:05:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:05:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:05:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:05:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:05:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:05:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:58.130973+00:00 (in 29.999505 seconds)
buildlogsarchiver stdout | 2024-04-22 04:05:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:58 UTC)" (scheduled at 2024-04-22 04:05:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:05:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 5, 28, 131675), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:05:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:05:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:05:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:05:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:31.933721+00:00 (in 2.002149 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:29 UTC)" (scheduled at 2024-04-22 04:05:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,933 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 29, 932029), True, datetime.datetime(2024, 4, 22, 4, 5, 29, 932029), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:05:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:05:30,633 [286] [DEBUG] [app] Starting request: urn:request:2d4df166-871b-423e-bdb9-be3c276938ae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:30,633 [285] [DEBUG] [app] Starting request: urn:request:6b4324b7-cb21-4cbb-bbf5-95616bec82e1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:30,635 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:30,635 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:05:30,637 [275] [DEBUG] [app] Starting request: urn:request:d600a431-c9cb-4fea-8981-1dbc00ab144d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:30,637 [283] [DEBUG] [app] Starting request: urn:request:87558965-034d-4fa9-ae18-5936785779a3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:30,637 [275] [DEBUG] [app] Ending request: urn:request:d600a431-c9cb-4fea-8981-1dbc00ab144d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d600a431-c9cb-4fea-8981-1dbc00ab144d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:30,637 [283] [DEBUG] [app] Ending request: urn:request:87558965-034d-4fa9-ae18-5936785779a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:87558965-034d-4fa9-ae18-5936785779a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:30,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:05:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:30,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:30,639 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:30,640 [285] [DEBUG] [app] Starting request: urn:request:b47580d3-88cc-48e9-96c9-a983c0659b73 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:30,640 [285] [DEBUG] [app] Ending request: urn:request:b47580d3-88cc-48e9-96c9-a983c0659b73 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b47580d3-88cc-48e9-96c9-a983c0659b73', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:30,640 [282] [DEBUG] [app] Starting request: urn:request:e80bef8e-2f37-4c1e-aee2-c9da3450de16 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:30,640 [282] [DEBUG] [app] Ending request: urn:request:e80bef8e-2f37-4c1e-aee2-c9da3450de16 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e80bef8e-2f37-4c1e-aee2-c9da3450de16', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:05:30,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:30,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:30,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:30,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:30,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:30,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:30,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:30,661 [285] [DEBUG] [app] Ending request: urn:request:6b4324b7-cb21-4cbb-bbf5-95616bec82e1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6b4324b7-cb21-4cbb-bbf5-95616bec82e1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:30,662 [286] [DEBUG] [app] Ending request: urn:request:2d4df166-871b-423e-bdb9-be3c276938ae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2d4df166-871b-423e-bdb9-be3c276938ae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:30,662 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:30,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:30,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:05:30,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:05:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:05:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:01.842666+00:00 (in 29.999479 seconds)
gcworker stdout | 2024-04-22 04:05:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:01 UTC)" (scheduled at 2024-04-22 04:05:31.842666+00:00)
gcworker stdout | 2024-04-22 04:05:31,843 [67] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2024-04-22 04:05:31,857 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549131856, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:05:31,863 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:05:31,863 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:05:31,863 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:05:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:05:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:29.931002+00:00 (in 57.996768 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:05:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:31 UTC)" (scheduled at 2024-04-22 04:05:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:05:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:05:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:05:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:42.430973+00:00 (in 9.999492 seconds)
notificationworker stdout | 2024-04-22 04:05:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:42 UTC)" (scheduled at 2024-04-22 04:05:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:05:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:05:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 32, 431765), True, datetime.datetime(2024, 4, 22, 4, 5, 32, 431765), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:05:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:05:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:05:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:05:35,217 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:05:36,503 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:05:39,089 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:05:40,138 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:05:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:05:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:10.238173+00:00 (in 29.999543 seconds)
autopruneworker stdout | 2024-04-22 04:05:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:10 UTC)" (scheduled at 2024-04-22 04:05:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:05:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758680247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:05:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:05:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:05:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:05:40,437 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:05:40,535 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:05:41,141 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:05:41,803 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:05:42,009 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:05:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:52.430973+00:00 (in 9.999476 seconds)
notificationworker stdout | 2024-04-22 04:05:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:52 UTC)" (scheduled at 2024-04-22 04:05:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:05:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:05:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 42, 431807), True, datetime.datetime(2024, 4, 22, 4, 5, 42, 431807), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:05:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:05:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:05:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:05:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:05:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:05:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:57.806778+00:00 (in 14.999454 seconds)
quotatotalworker stdout | 2024-04-22 04:05:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:57 UTC)" (scheduled at 2024-04-22 04:05:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:05:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:05:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:05:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:05:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:05:43,317 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:05:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:05:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:47.065745+00:00 (in 2.998149 seconds)
securityworker stdout | 2024-04-22 04:05:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:14 UTC)" (scheduled at 2024-04-22 04:05:44.067050+00:00)
securityworker stdout | 2024-04-22 04:05:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:05:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:05:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:05:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:05:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:05:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:05:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:05:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:05:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:05:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:05:44,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 0, 44, 77510), 1, 2])
securityworker stdout | 2024-04-22 04:05:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:05:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:05:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 0, 44, 77510), 1, 2])
securityworker stdout | 2024-04-22 04:05:44,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:05:44,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:05:44,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:05:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:05:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:05:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:05:44,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:05:44,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:05:44,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:05:44,269 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:05:45,632 [282] [DEBUG] [app] Starting request: urn:request:6797a8db-82c2-4c68-a2b2-4e33f2dd963b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:45,632 [286] [DEBUG] [app] Starting request: urn:request:3a4e87b6-25ae-494c-90cd-9961126de413 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:05:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:05:45,635 [278] [DEBUG] [app] Starting request: urn:request:70a9bf1d-7e96-4efd-9ab2-61a69d5ab62a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:45,635 [275] [DEBUG] [app] Starting request: urn:request:edbeefe0-29d7-4058-9dc2-3a935a8f929f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:05:45,636 [278] [DEBUG] [app] Ending request: urn:request:70a9bf1d-7e96-4efd-9ab2-61a69d5ab62a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:70a9bf1d-7e96-4efd-9ab2-61a69d5ab62a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:05:45,636 [275] [DEBUG] [app] Ending request: urn:request:edbeefe0-29d7-4058-9dc2-3a935a8f929f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:edbeefe0-29d7-4058-9dc2-3a935a8f929f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:05:45,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:05:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:05:45,639 [282] [DEBUG] [app] Starting request: urn:request:68f89f92-9a39-47e8-83e8-191a31402ec4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:45,639 [282] [DEBUG] [app] Ending request: urn:request:68f89f92-9a39-47e8-83e8-191a31402ec4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68f89f92-9a39-47e8-83e8-191a31402ec4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:45,639 [285] [DEBUG] [app] Starting request: urn:request:ae1dff65-6e8c-4e08-a15a-07ef56538938 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:05:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:05:45,639 [285] [DEBUG] [app] Ending request: urn:request:ae1dff65-6e8c-4e08-a15a-07ef56538938 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ae1dff65-6e8c-4e08-a15a-07ef56538938', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:05:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:05:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:05:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:05:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:05:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:05:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:05:45,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:05:45,659 [282] [DEBUG] [app] Ending request: urn:request:6797a8db-82c2-4c68-a2b2-4e33f2dd963b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6797a8db-82c2-4c68-a2b2-4e33f2dd963b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:05:45,660 [286] [DEBUG] [app] Ending request: urn:request:3a4e87b6-25ae-494c-90cd-9961126de413 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a4e87b6-25ae-494c-90cd-9961126de413', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:05:45,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:05:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:05:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:05:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:05:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:05:46,744 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:05:46,967 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:05:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:05:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:14.067050+00:00 (in 27.000750 seconds)
securityworker stdout | 2024-04-22 04:05:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:17 UTC)" (scheduled at 2024-04-22 04:05:47.065745+00:00)
securityworker stdout | 2024-04-22 04:05:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:05:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:05:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:05:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:05:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:05:47,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:05:47,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:05:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:05:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:10.248668+00:00 (in 23.002240 seconds)
namespacegcworker stdout | 2024-04-22 04:05:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:47 UTC)" (scheduled at 2024-04-22 04:05:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:05:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:05:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 47, 246673), True, datetime.datetime(2024, 4, 22, 4, 5, 47, 246673), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:05:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:05:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:05:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:05:47,446 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:05:48,065 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:05:49,091 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:05:49,840 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:05:49,843 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:05:49,845 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:05:50,071 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:05:50,534 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:05:51,348 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:05:51,414 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:05:51,416 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:05:51,421 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:05:51,429 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:05:51,432 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:05:51,441 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:05:51,448 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:05:51,451 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:05:51,514 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:05:51,523 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:05:51,525 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:05:51,528 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:05:51,532 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:05:51,767 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:05:52,056 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:05:52,220 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:05:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:05:54.433683+00:00 (in 2.002181 seconds)
notificationworker stdout | 2024-04-22 04:05:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:02 UTC)" (scheduled at 2024-04-22 04:05:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:05:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:05:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 52, 431806), True, datetime.datetime(2024, 4, 22, 4, 5, 52, 431806), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:05:52,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:05:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:05:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:02 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:05:52,989 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:05:53,033 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
exportactionlogsworker stdout | 2024-04-22 04:05:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:05:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:18.047111+00:00 (in 25.000978 seconds)
exportactionlogsworker stdout | 2024-04-22 04:05:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:53 UTC)" (scheduled at 2024-04-22 04:05:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:05:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:05:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 5, 53, 46563), True, datetime.datetime(2024, 4, 22, 4, 5, 53, 46563), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:05:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:05:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:05:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:53 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:05:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:05:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:02.430973+00:00 (in 7.996756 seconds)
notificationworker stdout | 2024-04-22 04:05:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:54 UTC)" (scheduled at 2024-04-22 04:05:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:05:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:05:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:06:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:05:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:05:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:12.806778+00:00 (in 14.999479 seconds)
quotatotalworker stdout | 2024-04-22 04:05:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:12 UTC)" (scheduled at 2024-04-22 04:05:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:05:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:05:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:05:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:05:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:05:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:28.130973+00:00 (in 29.999543 seconds)
buildlogsarchiver stdout | 2024-04-22 04:05:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:28 UTC)" (scheduled at 2024-04-22 04:05:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:05:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 5, 58, 131648), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:05:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:05:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:05:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:06:00,632 [285] [DEBUG] [app] Starting request: urn:request:ee658cbd-e14a-4bb3-9fd6-c137e9094d79 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:00,632 [286] [DEBUG] [app] Starting request: urn:request:56bb0b0b-1361-473d-9252-459b27ae92f2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:06:00,635 [279] [DEBUG] [app] Starting request: urn:request:9efbad1e-1660-4a78-8c9c-1c6fd1c6866d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:00,635 [283] [DEBUG] [app] Starting request: urn:request:f346d6fc-3fd8-43b8-b765-a2120dbd49b1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:00,636 [279] [DEBUG] [app] Ending request: urn:request:9efbad1e-1660-4a78-8c9c-1c6fd1c6866d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9efbad1e-1660-4a78-8c9c-1c6fd1c6866d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:00,636 [283] [DEBUG] [app] Ending request: urn:request:f346d6fc-3fd8-43b8-b765-a2120dbd49b1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f346d6fc-3fd8-43b8-b765-a2120dbd49b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:00,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:06:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:06:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:00,639 [286] [DEBUG] [app] Starting request: urn:request:12e47eb1-4f03-4801-a7ef-38739565e5c8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:00,639 [286] [DEBUG] [app] Ending request: urn:request:12e47eb1-4f03-4801-a7ef-38739565e5c8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:12e47eb1-4f03-4801-a7ef-38739565e5c8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:00,639 [282] [DEBUG] [app] Starting request: urn:request:78778097-845f-4abb-9540-bc76d07fb7c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:06:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:06:00,639 [282] [DEBUG] [app] Ending request: urn:request:78778097-845f-4abb-9540-bc76d07fb7c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:78778097-845f-4abb-9540-bc76d07fb7c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:00,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:06:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:06:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:00,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:00,660 [286] [DEBUG] [app] Ending request: urn:request:56bb0b0b-1361-473d-9252-459b27ae92f2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:56bb0b0b-1361-473d-9252-459b27ae92f2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:00,660 [285] [DEBUG] [app] Ending request: urn:request:ee658cbd-e14a-4bb3-9fd6-c137e9094d79 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ee658cbd-e14a-4bb3-9fd6-c137e9094d79', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:00,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:06:00,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:06:00,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:06:00,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:06:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:06:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:31.842666+00:00 (in 29.999398 seconds)
gcworker stdout | 2024-04-22 04:06:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:31 UTC)" (scheduled at 2024-04-22 04:06:01.842666+00:00)
gcworker stdout | 2024-04-22 04:06:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549161843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:06:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:06:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:06:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:06:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:06:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:09.033206+00:00 (in 6.997136 seconds)
repositorygcworker stdout | 2024-04-22 04:06:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:02 UTC)" (scheduled at 2024-04-22 04:06:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:06:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:06:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:06:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:12.430973+00:00 (in 9.999512 seconds)
notificationworker stdout | 2024-04-22 04:06:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:12 UTC)" (scheduled at 2024-04-22 04:06:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:06:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:06:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 2, 431767), True, datetime.datetime(2024, 4, 22, 4, 6, 2, 431767), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:06:02,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:06:02,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:06:02,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:06:05,254 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:06:06,539 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:06:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:06:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:02.035613+00:00 (in 53.001905 seconds)
repositorygcworker stdout | 2024-04-22 04:06:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:09 UTC)" (scheduled at 2024-04-22 04:06:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:06:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:06:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 9, 33949), True, datetime.datetime(2024, 4, 22, 4, 6, 9, 33949), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:06:09,047 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:06:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:06:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:06:09,124 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:06:10,151 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:06:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:06:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:40.238173+00:00 (in 29.999458 seconds)
autopruneworker stdout | 2024-04-22 04:06:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:40 UTC)" (scheduled at 2024-04-22 04:06:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:06:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758710247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:06:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:06:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:30.247852+00:00 (in 19.998719 seconds)
namespacegcworker stdout | 2024-04-22 04:06:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:10 UTC)" (scheduled at 2024-04-22 04:06:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:06:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:06:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:06:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:06:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:06:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:06:10,474 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:06:10,558 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:06:11,148 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:06:11,815 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:06:12,045 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:06:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:22.430973+00:00 (in 9.999496 seconds)
notificationworker stdout | 2024-04-22 04:06:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:22 UTC)" (scheduled at 2024-04-22 04:06:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:06:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:06:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 12, 431737), True, datetime.datetime(2024, 4, 22, 4, 6, 12, 431737), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:06:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:06:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:06:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:06:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:06:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:27.806778+00:00 (in 14.999478 seconds)
quotatotalworker stdout | 2024-04-22 04:06:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:27 UTC)" (scheduled at 2024-04-22 04:06:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:06:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:06:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:06:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:06:13,324 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:06:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:06:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:17.065745+00:00 (in 2.998133 seconds)
securityworker stdout | 2024-04-22 04:06:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:44 UTC)" (scheduled at 2024-04-22 04:06:14.067050+00:00)
securityworker stdout | 2024-04-22 04:06:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:06:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:06:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:06:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:06:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:06:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:06:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:06:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:06:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:06:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:06:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 1, 14, 77471), 1, 2])
securityworker stdout | 2024-04-22 04:06:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:06:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:06:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 1, 14, 77471), 1, 2])
securityworker stdout | 2024-04-22 04:06:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:06:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:06:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:06:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:06:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:06:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:06:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:06:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:06:14,305 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:06:15,632 [282] [DEBUG] [app] Starting request: urn:request:29beb8ee-cb79-4361-910b-a653628b09c3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:15,632 [286] [DEBUG] [app] Starting request: urn:request:5968c742-a6d3-4595-a451-ad5f135186e9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:06:15,636 [283] [DEBUG] [app] Starting request: urn:request:5d7a1bd5-9c73-4d43-8136-6c84468b74c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:15,636 [281] [DEBUG] [app] Starting request: urn:request:2a6a09f8-c914-4779-8e9c-33e3d1de5706 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:15,636 [283] [DEBUG] [app] Ending request: urn:request:5d7a1bd5-9c73-4d43-8136-6c84468b74c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d7a1bd5-9c73-4d43-8136-6c84468b74c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:15,636 [281] [DEBUG] [app] Ending request: urn:request:2a6a09f8-c914-4779-8e9c-33e3d1de5706 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a6a09f8-c914-4779-8e9c-33e3d1de5706', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:06:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:06:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:15,639 [286] [DEBUG] [app] Starting request: urn:request:73bfb4a3-ac86-43c8-8ff9-335e78b9894c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:15,639 [286] [DEBUG] [app] Ending request: urn:request:73bfb4a3-ac86-43c8-8ff9-335e78b9894c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:73bfb4a3-ac86-43c8-8ff9-335e78b9894c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:06:15,639 [285] [DEBUG] [app] Starting request: urn:request:c52f2949-f291-4b7e-aba0-c94e5ca2a339 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:15,640 [285] [DEBUG] [app] Ending request: urn:request:c52f2949-f291-4b7e-aba0-c94e5ca2a339 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c52f2949-f291-4b7e-aba0-c94e5ca2a339', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:06:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:06:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:15,660 [282] [DEBUG] [app] Ending request: urn:request:29beb8ee-cb79-4361-910b-a653628b09c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:29beb8ee-cb79-4361-910b-a653628b09c3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:06:15,660 [286] [DEBUG] [app] Ending request: urn:request:5968c742-a6d3-4595-a451-ad5f135186e9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5968c742-a6d3-4595-a451-ad5f135186e9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:06:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:06:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
expiredappspecifictokenworker stdout | 2024-04-22 04:06:16,778 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:06:17,003 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:06:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:06:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:44.067050+00:00 (in 27.000765 seconds)
securityworker stdout | 2024-04-22 04:06:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:47 UTC)" (scheduled at 2024-04-22 04:06:17.065745+00:00)
securityworker stdout | 2024-04-22 04:06:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:06:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:06:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:06:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:06:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:06:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:06:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:06:17,464 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:06:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:06:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:53.045554+00:00 (in 34.997855 seconds)
exportactionlogsworker stdout | 2024-04-22 04:06:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:18 UTC)" (scheduled at 2024-04-22 04:06:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:06:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:06:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:06:18,101 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:06:19,102 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:06:19,849 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:06:19,852 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:06:19,855 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:06:20,109 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:06:20,553 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:06:21,361 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:06:21,421 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:06:21,429 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:06:21,431 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:06:21,437 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:06:21,440 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:06:21,452 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:06:21,455 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:06:21,461 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:06:21,521 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:06:21,530 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:06:21,532 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:06:21,535 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:06:21,539 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:06:21,803 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:06:22,092 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:06:22,257 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:06:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:32.430973+00:00 (in 9.999448 seconds)
notificationworker stdout | 2024-04-22 04:06:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:32 UTC)" (scheduled at 2024-04-22 04:06:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:06:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:06:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 22, 431854), True, datetime.datetime(2024, 4, 22, 4, 6, 22, 431854), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:06:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:06:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:06:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:06:23,025 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:06:23,068 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:06:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:06:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:24.738365+00:00 (in 59.999460 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:06:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:24 UTC)" (scheduled at 2024-04-22 04:06:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:06:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:06:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:06:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:06:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:06:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:42.806778+00:00 (in 14.999497 seconds)
quotatotalworker stdout | 2024-04-22 04:06:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:42 UTC)" (scheduled at 2024-04-22 04:06:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:06:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:06:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:06:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:06:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:06:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:58.130973+00:00 (in 29.999498 seconds)
buildlogsarchiver stdout | 2024-04-22 04:06:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:58 UTC)" (scheduled at 2024-04-22 04:06:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:06:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 6, 28, 131713), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:06:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:06:28,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:06:28,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:06:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:31.933721+00:00 (in 2.002178 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:29 UTC)" (scheduled at 2024-04-22 04:06:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 29, 931959), True, datetime.datetime(2024, 4, 22, 4, 6, 29, 931959), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:06:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:06:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:06:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:47.245913+00:00 (in 16.997563 seconds)
namespacegcworker stdout | 2024-04-22 04:06:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:11:30 UTC)" (scheduled at 2024-04-22 04:06:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 04:06:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 6, 30, 248555), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:06:30,262 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 6, 30, 248555), True, datetime.datetime(2024, 4, 22, 4, 6, 30, 248555), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:06:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 6, 30, 248555), True, datetime.datetime(2024, 4, 22, 4, 6, 30, 248555), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 4, 6, 30, 248555), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:06:30,270 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:06:30,270 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:11:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:06:30,632 [282] [DEBUG] [app] Starting request: urn:request:dcc44486-4bae-4e23-896a-16abab1b9691 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:30,632 [285] [DEBUG] [app] Starting request: urn:request:09d1537d-e0f8-4c79-95c5-4a3da71cbb35 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:06:30,635 [281] [DEBUG] [app] Starting request: urn:request:2a1faf9e-db82-4543-b450-4c20965aaf04 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:30,635 [275] [DEBUG] [app] Starting request: urn:request:5d477790-cb86-4641-a37b-68f055b73ce7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:30,636 [281] [DEBUG] [app] Ending request: urn:request:2a1faf9e-db82-4543-b450-4c20965aaf04 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a1faf9e-db82-4543-b450-4c20965aaf04', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:30,636 [275] [DEBUG] [app] Ending request: urn:request:5d477790-cb86-4641-a37b-68f055b73ce7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d477790-cb86-4641-a37b-68f055b73ce7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:06:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:06:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:30,639 [282] [DEBUG] [app] Starting request: urn:request:c26dfa36-3e41-44da-8540-e2f6616f2545 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:30,639 [282] [DEBUG] [app] Ending request: urn:request:c26dfa36-3e41-44da-8540-e2f6616f2545 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c26dfa36-3e41-44da-8540-e2f6616f2545', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:30,639 [284] [DEBUG] [app] Starting request: urn:request:30a7d7b5-a90f-4df8-b751-9a6cfb4aba70 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:06:30,639 [284] [DEBUG] [app] Ending request: urn:request:30a7d7b5-a90f-4df8-b751-9a6cfb4aba70 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:30a7d7b5-a90f-4df8-b751-9a6cfb4aba70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:06:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:06:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:30,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:30,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:30,659 [285] [DEBUG] [app] Ending request: urn:request:09d1537d-e0f8-4c79-95c5-4a3da71cbb35 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:09d1537d-e0f8-4c79-95c5-4a3da71cbb35', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:30,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:06:30,660 [282] [DEBUG] [app] Ending request: urn:request:dcc44486-4bae-4e23-896a-16abab1b9691 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:dcc44486-4bae-4e23-896a-16abab1b9691', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:30,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:06:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.028)
gunicorn-web stdout | 2024-04-22 04:06:30,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:06:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:06:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:01.842666+00:00 (in 29.999409 seconds)
gcworker stdout | 2024-04-22 04:06:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:01 UTC)" (scheduled at 2024-04-22 04:06:31.842666+00:00)
gcworker stdout | 2024-04-22 04:06:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549191843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:06:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:06:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:06:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:06:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:06:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:29.931002+00:00 (in 57.996776 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:06:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:31 UTC)" (scheduled at 2024-04-22 04:06:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:06:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:06:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:06:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:42.430973+00:00 (in 9.999464 seconds)
notificationworker stdout | 2024-04-22 04:06:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:42 UTC)" (scheduled at 2024-04-22 04:06:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:06:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:06:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 32, 431789), True, datetime.datetime(2024, 4, 22, 4, 6, 32, 431789), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:06:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:06:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:06:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:06:35,268 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:06:36,549 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:06:39,147 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:06:40,159 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:06:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:06:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:10.238173+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2024-04-22 04:06:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:10 UTC)" (scheduled at 2024-04-22 04:06:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:06:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758740247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:06:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:06:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:06:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:06:40,484 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:06:40,595 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:06:41,165 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:06:41,829 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:06:42,082 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:06:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:52.430973+00:00 (in 9.999456 seconds)
notificationworker stdout | 2024-04-22 04:06:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:52 UTC)" (scheduled at 2024-04-22 04:06:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:06:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:06:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 42, 431809), True, datetime.datetime(2024, 4, 22, 4, 6, 42, 431809), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:06:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:06:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:06:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:06:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:06:42,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:06:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:57.806778+00:00 (in 14.999499 seconds)
quotatotalworker stdout | 2024-04-22 04:06:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:57 UTC)" (scheduled at 2024-04-22 04:06:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:06:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:06:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:06:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:06:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:06:43,361 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:06:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:06:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:47.065745+00:00 (in 2.998196 seconds)
securityworker stdout | 2024-04-22 04:06:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:14 UTC)" (scheduled at 2024-04-22 04:06:44.067050+00:00)
securityworker stdout | 2024-04-22 04:06:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:06:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:06:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:06:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:06:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:06:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:06:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:06:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:06:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:06:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:06:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 1, 44, 78291), 1, 2])
securityworker stdout | 2024-04-22 04:06:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:06:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:06:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 1, 44, 78291), 1, 2])
securityworker stdout | 2024-04-22 04:06:44,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:06:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:06:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:06:44,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:06:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:06:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:06:44,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:06:44,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:06:44,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:06:44,317 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:06:45,632 [285] [DEBUG] [app] Starting request: urn:request:98c28670-0f46-4cfc-9f98-3077563073a2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:45,632 [286] [DEBUG] [app] Starting request: urn:request:8735de26-0ae6-4967-9315-d11c0bc96004 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:06:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:06:45,636 [283] [DEBUG] [app] Starting request: urn:request:d3547ebd-dfdb-4af6-919c-4431255a21c2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:45,636 [281] [DEBUG] [app] Starting request: urn:request:1c3495a7-6554-4ce9-adf4-bb8eb7821c62 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:06:45,636 [283] [DEBUG] [app] Ending request: urn:request:d3547ebd-dfdb-4af6-919c-4431255a21c2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d3547ebd-dfdb-4af6-919c-4431255a21c2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:45,636 [281] [DEBUG] [app] Ending request: urn:request:1c3495a7-6554-4ce9-adf4-bb8eb7821c62 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1c3495a7-6554-4ce9-adf4-bb8eb7821c62', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:06:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:06:45,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:06:45,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:45,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:06:45,639 [286] [DEBUG] [app] Starting request: urn:request:8094e68e-08f9-4de3-8b24-f6eb23286a1e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:45,639 [286] [DEBUG] [app] Ending request: urn:request:8094e68e-08f9-4de3-8b24-f6eb23286a1e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8094e68e-08f9-4de3-8b24-f6eb23286a1e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:45,639 [282] [DEBUG] [app] Starting request: urn:request:0c532969-a789-4188-9fc5-2299be7bab7f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:06:45,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:06:45,640 [282] [DEBUG] [app] Ending request: urn:request:0c532969-a789-4188-9fc5-2299be7bab7f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0c532969-a789-4188-9fc5-2299be7bab7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:06:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:06:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:06:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:06:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:06:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:45,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:06:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:45,656 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:06:45,660 [286] [DEBUG] [app] Ending request: urn:request:8735de26-0ae6-4967-9315-d11c0bc96004 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8735de26-0ae6-4967-9315-d11c0bc96004', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:06:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:06:45,667 [285] [DEBUG] [app] Ending request: urn:request:98c28670-0f46-4cfc-9f98-3077563073a2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:98c28670-0f46-4cfc-9f98-3077563073a2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:06:45,667 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:06:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.036)
gunicorn-web stdout | 2024-04-22 04:06:45,667 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:06:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 04:06:46,814 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:06:47,029 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:06:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:06:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:14.067050+00:00 (in 27.000763 seconds)
securityworker stdout | 2024-04-22 04:06:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:17 UTC)" (scheduled at 2024-04-22 04:06:47.065745+00:00)
securityworker stdout | 2024-04-22 04:06:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:06:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:06:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:06:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:06:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:06:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:06:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:06:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:06:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:10.248668+00:00 (in 23.002225 seconds)
namespacegcworker stdout | 2024-04-22 04:06:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:47 UTC)" (scheduled at 2024-04-22 04:06:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:06:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:06:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 47, 246700), True, datetime.datetime(2024, 4, 22, 4, 6, 47, 246700), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:06:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:06:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:06:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:06:47,500 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:06:48,140 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:06:49,125 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:06:49,856 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:06:49,859 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:06:49,863 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:06:50,145 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:06:50,568 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:06:51,372 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:06:51,429 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:06:51,441 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:06:51,443 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:06:51,445 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:06:51,450 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:06:51,466 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:06:51,468 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:06:51,470 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:06:51,529 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:06:51,537 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:06:51,540 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:06:51,542 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:06:51,547 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:06:51,829 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:06:52,108 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:06:52,293 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:06:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:06:54.433683+00:00 (in 2.002234 seconds)
notificationworker stdout | 2024-04-22 04:06:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:02 UTC)" (scheduled at 2024-04-22 04:06:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:06:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:06:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 52, 431720), True, datetime.datetime(2024, 4, 22, 4, 6, 52, 431720), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:06:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:06:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:06:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:06:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:06:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:18.047111+00:00 (in 25.001070 seconds)
exportactionlogsworker stdout | 2024-04-22 04:06:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:53 UTC)" (scheduled at 2024-04-22 04:06:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:06:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:06:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 6, 53, 46466), True, datetime.datetime(2024, 4, 22, 4, 6, 53, 46466), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:06:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:06:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:06:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:06:53,062 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:06:53,096 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:06:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:06:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:02.430973+00:00 (in 7.996770 seconds)
notificationworker stdout | 2024-04-22 04:06:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:54 UTC)" (scheduled at 2024-04-22 04:06:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:06:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:06:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:07:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:06:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:06:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:12.806778+00:00 (in 14.999526 seconds)
quotatotalworker stdout | 2024-04-22 04:06:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:12 UTC)" (scheduled at 2024-04-22 04:06:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:06:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:06:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:06:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:06:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:06:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:28.130973+00:00 (in 29.999496 seconds)
buildlogsarchiver stdout | 2024-04-22 04:06:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:28 UTC)" (scheduled at 2024-04-22 04:06:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:06:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 6, 58, 131727), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:06:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:06:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:06:58,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:07:00,632 [286] [DEBUG] [app] Starting request: urn:request:132b912e-4fcf-412c-b87d-8014284264f9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:00,632 [282] [DEBUG] [app] Starting request: urn:request:feb68df9-748d-4fbe-81bd-9d25f5fdb17a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:07:00,635 [276] [DEBUG] [app] Starting request: urn:request:326b599e-33bc-4f11-ab4f-b45e2143a0a0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:00,635 [278] [DEBUG] [app] Starting request: urn:request:9843b1ed-cf1d-4847-b75f-087d3210c5d9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:00,636 [278] [DEBUG] [app] Ending request: urn:request:9843b1ed-cf1d-4847-b75f-087d3210c5d9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9843b1ed-cf1d-4847-b75f-087d3210c5d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:07:00,636 [276] [DEBUG] [app] Ending request: urn:request:326b599e-33bc-4f11-ab4f-b45e2143a0a0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:326b599e-33bc-4f11-ab4f-b45e2143a0a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:07:00,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:07:00,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:07:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:00,639 [282] [DEBUG] [app] Starting request: urn:request:303c7ac6-20b4-45db-bbfd-dc13f05b8bb0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:00,639 [284] [DEBUG] [app] Starting request: urn:request:656b1a03-7ef3-4c87-8b59-9c29b1ae64ba (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:00,639 [282] [DEBUG] [app] Ending request: urn:request:303c7ac6-20b4-45db-bbfd-dc13f05b8bb0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:303c7ac6-20b4-45db-bbfd-dc13f05b8bb0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:00,639 [284] [DEBUG] [app] Ending request: urn:request:656b1a03-7ef3-4c87-8b59-9c29b1ae64ba (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:656b1a03-7ef3-4c87-8b59-9c29b1ae64ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:07:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:00,661 [282] [DEBUG] [app] Ending request: urn:request:feb68df9-748d-4fbe-81bd-9d25f5fdb17a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:feb68df9-748d-4fbe-81bd-9d25f5fdb17a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:00,661 [286] [DEBUG] [app] Ending request: urn:request:132b912e-4fcf-412c-b87d-8014284264f9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:132b912e-4fcf-412c-b87d-8014284264f9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:07:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:07:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:07:00,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gcworker stdout | 2024-04-22 04:07:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:07:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:31.842666+00:00 (in 29.999455 seconds)
gcworker stdout | 2024-04-22 04:07:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:31 UTC)" (scheduled at 2024-04-22 04:07:01.842666+00:00)
gcworker stdout | 2024-04-22 04:07:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549221843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:07:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:07:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:07:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:07:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:07:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:09.033206+00:00 (in 6.997074 seconds)
repositorygcworker stdout | 2024-04-22 04:07:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:02 UTC)" (scheduled at 2024-04-22 04:07:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:07:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:07:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:07:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:12.430973+00:00 (in 9.999569 seconds)
notificationworker stdout | 2024-04-22 04:07:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:12 UTC)" (scheduled at 2024-04-22 04:07:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:07:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:07:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 2, 431794), True, datetime.datetime(2024, 4, 22, 4, 7, 2, 431794), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:07:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:07:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:07:05,304 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:07:06,586 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:07:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:07:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:02.035613+00:00 (in 53.001922 seconds)
repositorygcworker stdout | 2024-04-22 04:07:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:09 UTC)" (scheduled at 2024-04-22 04:07:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:07:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:07:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 9, 34007), True, datetime.datetime(2024, 4, 22, 4, 7, 9, 34007), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:07:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:07:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:07:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:07:09,183 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:07:10,184 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:07:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:07:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:40.238173+00:00 (in 29.999558 seconds)
autopruneworker stdout | 2024-04-22 04:07:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:40 UTC)" (scheduled at 2024-04-22 04:07:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:07:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758770247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:07:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:07:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:47.245913+00:00 (in 36.996750 seconds)
namespacegcworker stdout | 2024-04-22 04:07:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:10 UTC)" (scheduled at 2024-04-22 04:07:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:07:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:07:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:07:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:07:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:07:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:07:10,520 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:07:10,630 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:07:11,200 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:07:11,850 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:07:12,119 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:07:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:22.430973+00:00 (in 9.999498 seconds)
notificationworker stdout | 2024-04-22 04:07:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:22 UTC)" (scheduled at 2024-04-22 04:07:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:07:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:07:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 12, 431910), True, datetime.datetime(2024, 4, 22, 4, 7, 12, 431910), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:07:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:07:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:07:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:07:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:27.806778+00:00 (in 14.999460 seconds)
quotatotalworker stdout | 2024-04-22 04:07:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:27 UTC)" (scheduled at 2024-04-22 04:07:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:07:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:07:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:07:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:07:13,372 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:07:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:07:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:17.065745+00:00 (in 2.998201 seconds)
securityworker stdout | 2024-04-22 04:07:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:44 UTC)" (scheduled at 2024-04-22 04:07:14.067050+00:00)
securityworker stdout | 2024-04-22 04:07:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:07:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:07:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:07:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:07:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:07:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:07:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:07:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:07:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:07:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:07:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 2, 14, 77320), 1, 2])
securityworker stdout | 2024-04-22 04:07:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:07:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:07:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 2, 14, 77320), 1, 2])
securityworker stdout | 2024-04-22 04:07:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:07:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:07:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:07:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:07:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:07:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:07:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:07:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:07:14,324 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:07:15,632 [282] [DEBUG] [app] Starting request: urn:request:5b3e9ddb-69d3-4ad8-94eb-2be6cbb39242 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:15,632 [285] [DEBUG] [app] Starting request: urn:request:d1aaaafc-7342-4bc5-9a98-2685a5329d57 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:07:15,635 [275] [DEBUG] [app] Starting request: urn:request:9941022b-0731-4286-884a-80fd39aa7ab4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:15,635 [283] [DEBUG] [app] Starting request: urn:request:9d4abc46-5ac8-4a72-a73a-b5872a647878 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:15,636 [283] [DEBUG] [app] Ending request: urn:request:9d4abc46-5ac8-4a72-a73a-b5872a647878 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d4abc46-5ac8-4a72-a73a-b5872a647878', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:07:15,636 [275] [DEBUG] [app] Ending request: urn:request:9941022b-0731-4286-884a-80fd39aa7ab4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9941022b-0731-4286-884a-80fd39aa7ab4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:07:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:07:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 04:07:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:15,638 [282] [DEBUG] [app] Starting request: urn:request:f28bc1b1-c32c-4fdd-baed-17b728b2ff7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:15,639 [282] [DEBUG] [app] Ending request: urn:request:f28bc1b1-c32c-4fdd-baed-17b728b2ff7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f28bc1b1-c32c-4fdd-baed-17b728b2ff7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:15,639 [286] [DEBUG] [app] Starting request: urn:request:d735057d-69ba-417e-9e97-9f8e37bc336e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:07:15,639 [286] [DEBUG] [app] Ending request: urn:request:d735057d-69ba-417e-9e97-9f8e37bc336e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d735057d-69ba-417e-9e97-9f8e37bc336e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:07:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:15,659 [282] [DEBUG] [app] Ending request: urn:request:5b3e9ddb-69d3-4ad8-94eb-2be6cbb39242 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5b3e9ddb-69d3-4ad8-94eb-2be6cbb39242', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:15,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:07:15,659 [285] [DEBUG] [app] Ending request: urn:request:d1aaaafc-7342-4bc5-9a98-2685a5329d57 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1aaaafc-7342-4bc5-9a98-2685a5329d57', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:07:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:07:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:07:16,848 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:07:17,065 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
securityworker stdout | 2024-04-22 04:07:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:07:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:44.067050+00:00 (in 27.000793 seconds)
securityworker stdout | 2024-04-22 04:07:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:47 UTC)" (scheduled at 2024-04-22 04:07:17.065745+00:00)
securityworker stdout | 2024-04-22 04:07:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:07:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:07:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:07:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:07:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:07:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:07:17,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:07:17,525 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:07:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:07:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:23.046900+00:00 (in 4.999259 seconds)
exportactionlogsworker stdout | 2024-04-22 04:07:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:18 UTC)" (scheduled at 2024-04-22 04:07:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:07:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:07:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:07:18,175 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:07:19,138 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:07:19,865 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:07:19,867 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:07:19,870 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:07:20,163 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:07:20,598 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:07:21,383 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:07:21,439 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:07:21,451 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:07:21,453 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:07:21,455 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:07:21,459 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:07:21,477 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:07:21,479 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:07:21,482 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:07:21,535 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:07:21,543 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:07:21,548 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:07:21,550 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:07:21,553 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:07:21,847 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:07:22,126 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:07:22,303 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:07:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:32.430973+00:00 (in 9.999548 seconds)
notificationworker stdout | 2024-04-22 04:07:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:32 UTC)" (scheduled at 2024-04-22 04:07:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:07:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:07:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 22, 431813), True, datetime.datetime(2024, 4, 22, 4, 7, 22, 431813), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:07:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:07:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:32 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:07:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:07:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:53.045554+00:00 (in 29.998081 seconds)
exportactionlogsworker stdout | 2024-04-22 04:07:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:12:23 UTC)" (scheduled at 2024-04-22 04:07:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 04:07:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 7, 23, 47840), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:07:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 7, 23, 47840), True, datetime.datetime(2024, 4, 22, 4, 7, 23, 47840), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:07:23,065 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 7, 23, 47840), True, datetime.datetime(2024, 4, 22, 4, 7, 23, 47840), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 4, 7, 23, 47840), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:07:23,069 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:07:23,069 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:12:23 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:07:23,097 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:07:23,132 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:07:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:07:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:24.738365+00:00 (in 59.999470 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:07:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:24 UTC)" (scheduled at 2024-04-22 04:07:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:07:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:07:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:07:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:07:27,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:07:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:42.806778+00:00 (in 14.999450 seconds)
quotatotalworker stdout | 2024-04-22 04:07:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:42 UTC)" (scheduled at 2024-04-22 04:07:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:07:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:07:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:07:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:07:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:07:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:58.130973+00:00 (in 29.999456 seconds)
buildlogsarchiver stdout | 2024-04-22 04:07:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:58 UTC)" (scheduled at 2024-04-22 04:07:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:07:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 7, 28, 131753), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:07:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:07:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:07:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:07:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:31.933721+00:00 (in 2.002238 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:29 UTC)" (scheduled at 2024-04-22 04:07:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 29, 931737), True, datetime.datetime(2024, 4, 22, 4, 7, 29, 931737), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:07:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:07:30,632 [282] [DEBUG] [app] Starting request: urn:request:7acdd8dd-fe25-419e-a2c6-7890b1fbba48 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:30,632 [286] [DEBUG] [app] Starting request: urn:request:acef329c-5ef9-497f-afeb-4e241871bdd9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:07:30,635 [277] [DEBUG] [app] Starting request: urn:request:cc3d0a76-e752-438a-b259-0f13c6422d27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:30,635 [283] [DEBUG] [app] Starting request: urn:request:cd6d44ec-a46e-4f0e-a6f8-9a4f0d7180a3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:30,635 [283] [DEBUG] [app] Ending request: urn:request:cd6d44ec-a46e-4f0e-a6f8-9a4f0d7180a3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cd6d44ec-a46e-4f0e-a6f8-9a4f0d7180a3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:07:30,635 [277] [DEBUG] [app] Ending request: urn:request:cc3d0a76-e752-438a-b259-0f13c6422d27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cc3d0a76-e752-438a-b259-0f13c6422d27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:07:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:07:30,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:07:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:30,639 [284] [DEBUG] [app] Starting request: urn:request:bb8dbe9c-b815-47ca-a395-7400ab0e6418 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:30,639 [285] [DEBUG] [app] Starting request: urn:request:877e13e6-c79a-4739-9d70-24270be82f6a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:30,639 [285] [DEBUG] [app] Ending request: urn:request:877e13e6-c79a-4739-9d70-24270be82f6a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:877e13e6-c79a-4739-9d70-24270be82f6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:30,639 [284] [DEBUG] [app] Ending request: urn:request:bb8dbe9c-b815-47ca-a395-7400ab0e6418 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bb8dbe9c-b815-47ca-a395-7400ab0e6418', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:07:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:30,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:30,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:30,661 [282] [DEBUG] [app] Ending request: urn:request:7acdd8dd-fe25-419e-a2c6-7890b1fbba48 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7acdd8dd-fe25-419e-a2c6-7890b1fbba48', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:07:30,661 [286] [DEBUG] [app] Ending request: urn:request:acef329c-5ef9-497f-afeb-4e241871bdd9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:acef329c-5ef9-497f-afeb-4e241871bdd9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:30,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:07:30,662 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.031)
gunicorn-web stdout | 2024-04-22 04:07:30,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 04:07:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:07:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:01.842666+00:00 (in 29.999454 seconds)
gcworker stdout | 2024-04-22 04:07:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:01 UTC)" (scheduled at 2024-04-22 04:07:31.842666+00:00)
gcworker stdout | 2024-04-22 04:07:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549251843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:07:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:07:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:07:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:07:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:07:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:29.931002+00:00 (in 57.996789 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:07:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:31 UTC)" (scheduled at 2024-04-22 04:07:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:07:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:07:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:07:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:38.433394+00:00 (in 6.001931 seconds)
notificationworker stdout | 2024-04-22 04:07:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:42 UTC)" (scheduled at 2024-04-22 04:07:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:07:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:07:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 32, 431883), True, datetime.datetime(2024, 4, 22, 4, 7, 32, 431883), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:07:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:07:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:07:35,322 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:07:36,623 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
notificationworker stdout | 2024-04-22 04:07:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:42.430973+00:00 (in 3.997112 seconds)
notificationworker stdout | 2024-04-22 04:07:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:12:38 UTC)" (scheduled at 2024-04-22 04:07:38.433394+00:00)
notificationworker stdout | 2024-04-22 04:07:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 7, 38, 434249), 'notification/%'])
notificationworker stdout | 2024-04-22 04:07:38,447 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 7, 38, 434249), True, datetime.datetime(2024, 4, 22, 4, 7, 38, 434249), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 04:07:38,451 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 7, 38, 434249), True, datetime.datetime(2024, 4, 22, 4, 7, 38, 434249), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 4, 7, 38, 434249), 'notification/%'])
notificationworker stdout | 2024-04-22 04:07:38,455 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:38,455 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:12:38 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:07:39,220 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:07:40,202 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:07:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:07:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:10.238173+00:00 (in 29.999548 seconds)
autopruneworker stdout | 2024-04-22 04:07:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:10 UTC)" (scheduled at 2024-04-22 04:07:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:07:40,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758800247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:07:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:07:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:07:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:10 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:07:40,547 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:07:40,664 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:07:41,238 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:07:41,872 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:07:42,147 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:07:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:52.430973+00:00 (in 9.999509 seconds)
notificationworker stdout | 2024-04-22 04:07:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:52 UTC)" (scheduled at 2024-04-22 04:07:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:07:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:07:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 42, 431867), True, datetime.datetime(2024, 4, 22, 4, 7, 42, 431867), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:07:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:07:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:07:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:07:42,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:07:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:57.806778+00:00 (in 14.999509 seconds)
quotatotalworker stdout | 2024-04-22 04:07:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:57 UTC)" (scheduled at 2024-04-22 04:07:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:07:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:07:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:07:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:07:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:07:43,405 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:07:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:07:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:47.065745+00:00 (in 2.998118 seconds)
securityworker stdout | 2024-04-22 04:07:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:14 UTC)" (scheduled at 2024-04-22 04:07:44.067050+00:00)
securityworker stdout | 2024-04-22 04:07:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:07:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:07:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:07:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:07:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:07:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:07:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:07:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:07:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:07:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:07:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 2, 44, 77805), 1, 2])
securityworker stdout | 2024-04-22 04:07:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:07:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:07:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 2, 44, 77805), 1, 2])
securityworker stdout | 2024-04-22 04:07:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:07:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:07:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:07:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:07:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:07:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:07:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:07:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:07:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:07:44,360 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:07:45,632 [284] [DEBUG] [app] Starting request: urn:request:ecc3c3d4-1886-4e51-aa94-7eda347419cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:45,632 [282] [DEBUG] [app] Starting request: urn:request:d9b43350-3c2f-4a02-a2ae-65c0cd4153da (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:07:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:07:45,635 [283] [DEBUG] [app] Starting request: urn:request:78ed0939-ae38-4b0b-8dd2-b092f85edba9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:45,635 [280] [DEBUG] [app] Starting request: urn:request:d7dfdc1a-6042-48a9-a01f-54c899be389c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:07:45,636 [283] [DEBUG] [app] Ending request: urn:request:78ed0939-ae38-4b0b-8dd2-b092f85edba9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:78ed0939-ae38-4b0b-8dd2-b092f85edba9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:07:45,636 [280] [DEBUG] [app] Ending request: urn:request:d7dfdc1a-6042-48a9-a01f-54c899be389c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d7dfdc1a-6042-48a9-a01f-54c899be389c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:07:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:07:45,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:07:45,639 [282] [DEBUG] [app] Starting request: urn:request:a687b2ae-30a6-4c1d-b047-23bb1aa68dc7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:45,639 [286] [DEBUG] [app] Starting request: urn:request:b7150a80-d2d6-4fa2-8ee4-5922af60220d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:07:45,639 [282] [DEBUG] [app] Ending request: urn:request:a687b2ae-30a6-4c1d-b047-23bb1aa68dc7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a687b2ae-30a6-4c1d-b047-23bb1aa68dc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:45,639 [286] [DEBUG] [app] Ending request: urn:request:b7150a80-d2d6-4fa2-8ee4-5922af60220d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7150a80-d2d6-4fa2-8ee4-5922af60220d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:07:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:07:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:07:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:07:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:07:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:07:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:07:45,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:07:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:07:45,663 [284] [DEBUG] [app] Ending request: urn:request:ecc3c3d4-1886-4e51-aa94-7eda347419cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ecc3c3d4-1886-4e51-aa94-7eda347419cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:45,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:07:45,664 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
gunicorn-web stdout | 2024-04-22 04:07:45,665 [282] [DEBUG] [app] Ending request: urn:request:d9b43350-3c2f-4a02-a2ae-65c0cd4153da (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d9b43350-3c2f-4a02-a2ae-65c0cd4153da', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:07:45,665 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:07:45,665 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:07:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:07:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
expiredappspecifictokenworker stdout | 2024-04-22 04:07:46,874 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:07:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:07:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:14.067050+00:00 (in 27.000759 seconds)
securityworker stdout | 2024-04-22 04:07:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:17 UTC)" (scheduled at 2024-04-22 04:07:47.065745+00:00)
securityworker stdout | 2024-04-22 04:07:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:07:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:07:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:07:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:07:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:07:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:07:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:17 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:07:47,088 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
namespacegcworker stdout | 2024-04-22 04:07:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:07:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:10.248668+00:00 (in 23.002255 seconds)
namespacegcworker stdout | 2024-04-22 04:07:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:47 UTC)" (scheduled at 2024-04-22 04:07:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:07:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:07:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 47, 246686), True, datetime.datetime(2024, 4, 22, 4, 7, 47, 246686), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:07:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:07:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:07:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:07:47,547 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:07:48,212 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:07:49,150 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:07:49,874 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:07:49,876 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:07:49,879 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:07:50,201 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:07:50,634 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:07:51,393 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:07:51,449 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:07:51,461 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:07:51,464 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:07:51,467 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:07:51,469 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:07:51,486 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:07:51,489 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:07:51,492 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:07:51,542 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:07:51,550 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:07:51,554 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:07:51,557 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:07:51,560 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:07:51,874 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:07:52,162 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:07:52,318 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:07:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:07:54.433683+00:00 (in 2.002198 seconds)
notificationworker stdout | 2024-04-22 04:07:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:02 UTC)" (scheduled at 2024-04-22 04:07:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:07:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:07:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 52, 431895), True, datetime.datetime(2024, 4, 22, 4, 7, 52, 431895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:07:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:07:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:07:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:07:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:07:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:18.047111+00:00 (in 25.001046 seconds)
exportactionlogsworker stdout | 2024-04-22 04:07:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:53 UTC)" (scheduled at 2024-04-22 04:07:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:07:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:07:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 7, 53, 46485), True, datetime.datetime(2024, 4, 22, 4, 7, 53, 46485), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:07:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:07:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:07:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:07:53,134 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:07:53,167 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:07:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:07:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:02.430973+00:00 (in 7.996824 seconds)
notificationworker stdout | 2024-04-22 04:07:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:54 UTC)" (scheduled at 2024-04-22 04:07:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:07:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:07:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:08:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:07:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:07:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:12.806778+00:00 (in 14.999489 seconds)
quotatotalworker stdout | 2024-04-22 04:07:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:12 UTC)" (scheduled at 2024-04-22 04:07:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:07:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:07:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:07:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:07:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:07:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:28.130973+00:00 (in 29.999516 seconds)
buildlogsarchiver stdout | 2024-04-22 04:07:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:28 UTC)" (scheduled at 2024-04-22 04:07:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:07:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 7, 58, 131694), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:07:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:07:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:07:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:08:00,632 [284] [DEBUG] [app] Starting request: urn:request:310a829e-facf-4a4c-8023-8c51a83f054f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:00,632 [282] [DEBUG] [app] Starting request: urn:request:3e51f43b-7845-47c6-8ce0-ed8c288cec39 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:00,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:08:00,635 [281] [DEBUG] [app] Starting request: urn:request:07e303d5-dd8a-4f19-84e4-36f3c6adf8d2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:00,635 [275] [DEBUG] [app] Starting request: urn:request:3256fb8c-95e4-4337-9942-b5001bb100bd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:00,636 [275] [DEBUG] [app] Ending request: urn:request:3256fb8c-95e4-4337-9942-b5001bb100bd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3256fb8c-95e4-4337-9942-b5001bb100bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:00,636 [281] [DEBUG] [app] Ending request: urn:request:07e303d5-dd8a-4f19-84e4-36f3c6adf8d2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:07e303d5-dd8a-4f19-84e4-36f3c6adf8d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:08:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:00,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 04:08:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:00,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:00,639 [282] [DEBUG] [app] Starting request: urn:request:e1e3685e-fff4-4de6-8ee8-acf54020118e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:00,639 [285] [DEBUG] [app] Starting request: urn:request:1326303e-ad8a-4b55-885a-f58d4f754afe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:00,639 [282] [DEBUG] [app] Ending request: urn:request:e1e3685e-fff4-4de6-8ee8-acf54020118e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1e3685e-fff4-4de6-8ee8-acf54020118e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:08:00,639 [285] [DEBUG] [app] Ending request: urn:request:1326303e-ad8a-4b55-885a-f58d4f754afe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1326303e-ad8a-4b55-885a-f58d4f754afe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:08:00,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:00,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:08:00,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:08:00,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:00,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:00,658 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:00,660 [284] [DEBUG] [app] Ending request: urn:request:310a829e-facf-4a4c-8023-8c51a83f054f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:310a829e-facf-4a4c-8023-8c51a83f054f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:00,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:00,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 04:08:00,670 [282] [DEBUG] [app] Ending request: urn:request:3e51f43b-7845-47c6-8ce0-ed8c288cec39 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3e51f43b-7845-47c6-8ce0-ed8c288cec39', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:00,670 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:00,670 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.039 118 0.039)
gcworker stdout | 2024-04-22 04:08:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:08:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:31.842666+00:00 (in 29.999450 seconds)
gcworker stdout | 2024-04-22 04:08:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:31 UTC)" (scheduled at 2024-04-22 04:08:01.842666+00:00)
gcworker stdout | 2024-04-22 04:08:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549281843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:08:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:08:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:08:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:08:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:08:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:09.033206+00:00 (in 6.997034 seconds)
repositorygcworker stdout | 2024-04-22 04:08:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:02 UTC)" (scheduled at 2024-04-22 04:08:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:08:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:08:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:08:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:12.430973+00:00 (in 9.999586 seconds)
notificationworker stdout | 2024-04-22 04:08:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:12 UTC)" (scheduled at 2024-04-22 04:08:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:08:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:08:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 2, 431790), True, datetime.datetime(2024, 4, 22, 4, 8, 2, 431790), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:08:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:08:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:08:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:08:05,359 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:08:06,656 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:08:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:08:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:26.035364+00:00 (in 17.001688 seconds)
repositorygcworker stdout | 2024-04-22 04:08:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:09 UTC)" (scheduled at 2024-04-22 04:08:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:08:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:08:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 9, 33960), True, datetime.datetime(2024, 4, 22, 4, 8, 9, 33960), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:08:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:08:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:08:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:08:09,250 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:08:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:08:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:40.238173+00:00 (in 29.999559 seconds)
autopruneworker stdout | 2024-04-22 04:08:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:40 UTC)" (scheduled at 2024-04-22 04:08:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:08:10,239 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
autopruneworker stdout | 2024-04-22 04:08:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758830247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:08:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:08:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:47.245913+00:00 (in 36.996770 seconds)
namespacegcworker stdout | 2024-04-22 04:08:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:10 UTC)" (scheduled at 2024-04-22 04:08:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:08:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:08:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:08:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:08:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:08:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:40 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:08:10,581 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:08:10,699 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:08:11,274 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:08:11,894 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:08:12,175 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:08:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:22.430973+00:00 (in 9.999495 seconds)
notificationworker stdout | 2024-04-22 04:08:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:22 UTC)" (scheduled at 2024-04-22 04:08:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:08:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:08:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 12, 431909), True, datetime.datetime(2024, 4, 22, 4, 8, 12, 431909), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:08:12,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:08:12,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:08:12,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:08:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:08:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:27.806778+00:00 (in 14.999436 seconds)
quotatotalworker stdout | 2024-04-22 04:08:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:27 UTC)" (scheduled at 2024-04-22 04:08:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:08:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:08:12,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:08:12,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:08:13,437 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:08:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:08:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:17.065745+00:00 (in 2.998160 seconds)
securityworker stdout | 2024-04-22 04:08:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:44 UTC)" (scheduled at 2024-04-22 04:08:14.067050+00:00)
securityworker stdout | 2024-04-22 04:08:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:08:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:08:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:08:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:08:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:08:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:08:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:08:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:08:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:08:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:08:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 3, 14, 77822), 1, 2])
securityworker stdout | 2024-04-22 04:08:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:08:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:08:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 3, 14, 77822), 1, 2])
securityworker stdout | 2024-04-22 04:08:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:08:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:08:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:08:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:08:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:08:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:08:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:08:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:08:14,397 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:08:15,632 [286] [DEBUG] [app] Starting request: urn:request:5693c224-1a86-4ea8-8c48-5203b111c500 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:15,632 [282] [DEBUG] [app] Starting request: urn:request:43233782-443f-421b-b508-8cfe6c3ed2db (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:08:15,635 [281] [DEBUG] [app] Starting request: urn:request:b2b6301b-75d7-4e59-b55f-04afd60a694d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:15,635 [275] [DEBUG] [app] Starting request: urn:request:f811cdb2-0c7b-4033-9ea3-16456723a91a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:15,636 [281] [DEBUG] [app] Ending request: urn:request:b2b6301b-75d7-4e59-b55f-04afd60a694d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2b6301b-75d7-4e59-b55f-04afd60a694d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:15,636 [275] [DEBUG] [app] Ending request: urn:request:f811cdb2-0c7b-4033-9ea3-16456723a91a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f811cdb2-0c7b-4033-9ea3-16456723a91a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:08:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:15,639 [285] [DEBUG] [app] Starting request: urn:request:91308593-4e75-488a-a84c-9432180e2e7b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:15,639 [285] [DEBUG] [app] Ending request: urn:request:91308593-4e75-488a-a84c-9432180e2e7b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:91308593-4e75-488a-a84c-9432180e2e7b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:08:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:15,640 [282] [DEBUG] [app] Starting request: urn:request:831bb43e-6e1f-457f-9897-04971020a9ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:15,640 [282] [DEBUG] [app] Ending request: urn:request:831bb43e-6e1f-457f-9897-04971020a9ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:831bb43e-6e1f-457f-9897-04971020a9ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:08:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:15,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:15,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:15,661 [286] [DEBUG] [app] Ending request: urn:request:5693c224-1a86-4ea8-8c48-5203b111c500 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5693c224-1a86-4ea8-8c48-5203b111c500', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:15,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:08:15,664 [282] [DEBUG] [app] Ending request: urn:request:43233782-443f-421b-b508-8cfe6c3ed2db (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:43233782-443f-421b-b508-8cfe6c3ed2db', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:15,664 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:15,664 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.033 118 0.033)
expiredappspecifictokenworker stdout | 2024-04-22 04:08:16,910 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:08:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:08:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:44.067050+00:00 (in 27.000748 seconds)
securityworker stdout | 2024-04-22 04:08:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:47 UTC)" (scheduled at 2024-04-22 04:08:17.065745+00:00)
securityworker stdout | 2024-04-22 04:08:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:08:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:08:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:08:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:08:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:08:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:08:17,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:08:17,103 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:08:17,567 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:08:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:08:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:53.045554+00:00 (in 34.997950 seconds)
exportactionlogsworker stdout | 2024-04-22 04:08:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:18 UTC)" (scheduled at 2024-04-22 04:08:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:08:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:08:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:08:18,241 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:08:19,185 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:08:19,881 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:08:19,884 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:08:19,886 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:08:20,238 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:08:20,671 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:08:21,403 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:08:21,459 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:08:21,467 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:08:21,472 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:08:21,477 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:08:21,480 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:08:21,496 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:08:21,498 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:08:21,502 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:08:21,549 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:08:21,557 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:08:21,561 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:08:21,564 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:08:21,568 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:08:21,901 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:08:22,198 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:08:22,336 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:08:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:32.430973+00:00 (in 9.999500 seconds)
notificationworker stdout | 2024-04-22 04:08:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:32 UTC)" (scheduled at 2024-04-22 04:08:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:08:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:08:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 22, 431909), True, datetime.datetime(2024, 4, 22, 4, 8, 22, 431909), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:08:22,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:08:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:08:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:08:23,159 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:08:23,177 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:08:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:08:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:24.738365+00:00 (in 59.999467 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:08:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:24 UTC)" (scheduled at 2024-04-22 04:08:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:08:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:08:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:08:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:08:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:08:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:02.035613+00:00 (in 35.999761 seconds)
repositorygcworker stdout | 2024-04-22 04:08:26,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:13:26 UTC)" (scheduled at 2024-04-22 04:08:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 04:08:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 8, 26, 36298), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:08:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 8, 26, 36298), True, datetime.datetime(2024, 4, 22, 4, 8, 26, 36298), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:08:26,054 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 8, 26, 36298), True, datetime.datetime(2024, 4, 22, 4, 8, 26, 36298), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 4, 8, 26, 36298), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:08:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:08:26,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:13:26 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:08:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:08:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:42.806778+00:00 (in 14.999432 seconds)
quotatotalworker stdout | 2024-04-22 04:08:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:42 UTC)" (scheduled at 2024-04-22 04:08:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:08:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:08:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:08:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:08:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:08:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:58.130973+00:00 (in 29.999494 seconds)
buildlogsarchiver stdout | 2024-04-22 04:08:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:58 UTC)" (scheduled at 2024-04-22 04:08:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:08:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 8, 28, 131725), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:08:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:08:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:08:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:08:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:30.933445+00:00 (in 1.001948 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:29 UTC)" (scheduled at 2024-04-22 04:08:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 29, 931877), True, datetime.datetime(2024, 4, 22, 4, 8, 29, 931877), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:08:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:08:30,632 [282] [DEBUG] [app] Starting request: urn:request:ddd91916-dd25-4f44-b39b-35db9b08cf54 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:30,632 [284] [DEBUG] [app] Starting request: urn:request:572e1ddc-8ed9-4f5e-af6c-527b2d35a844 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:08:30,636 [283] [DEBUG] [app] Starting request: urn:request:5047d770-49a3-4c56-98aa-5c3a0064109f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:30,636 [275] [DEBUG] [app] Starting request: urn:request:fd42d6c6-feda-4fa5-8c8a-86ead0f2867c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:30,636 [275] [DEBUG] [app] Ending request: urn:request:fd42d6c6-feda-4fa5-8c8a-86ead0f2867c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fd42d6c6-feda-4fa5-8c8a-86ead0f2867c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:30,636 [283] [DEBUG] [app] Ending request: urn:request:5047d770-49a3-4c56-98aa-5c3a0064109f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5047d770-49a3-4c56-98aa-5c3a0064109f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:08:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:30,639 [282] [DEBUG] [app] Starting request: urn:request:af3090ce-d5ea-44c7-bc66-10e7c394e33b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:30,639 [282] [DEBUG] [app] Ending request: urn:request:af3090ce-d5ea-44c7-bc66-10e7c394e33b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af3090ce-d5ea-44c7-bc66-10e7c394e33b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:08:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:30,639 [285] [DEBUG] [app] Starting request: urn:request:76091e0e-383b-4f9f-9c49-1018e9124a89 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:30,640 [285] [DEBUG] [app] Ending request: urn:request:76091e0e-383b-4f9f-9c49-1018e9124a89 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76091e0e-383b-4f9f-9c49-1018e9124a89', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:08:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:08:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:30,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:30,659 [282] [DEBUG] [app] Ending request: urn:request:ddd91916-dd25-4f44-b39b-35db9b08cf54 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddd91916-dd25-4f44-b39b-35db9b08cf54', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:30,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.028)
gunicorn-web stdout | 2024-04-22 04:08:30,660 [284] [DEBUG] [app] Ending request: urn:request:572e1ddc-8ed9-4f5e-af6c-527b2d35a844 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:572e1ddc-8ed9-4f5e-af6c-527b2d35a844', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:08:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:31.933721+00:00 (in 0.999832 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:13:30 UTC)" (scheduled at 2024-04-22 04:08:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 8, 30, 934266), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,948 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 8, 30, 934266), True, datetime.datetime(2024, 4, 22, 4, 8, 30, 934266), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,952 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 8, 30, 934266), True, datetime.datetime(2024, 4, 22, 4, 8, 30, 934266), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 4, 8, 30, 934266), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,956 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:08:30,956 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:13:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:08:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:08:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:01.842666+00:00 (in 29.999482 seconds)
gcworker stdout | 2024-04-22 04:08:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:01 UTC)" (scheduled at 2024-04-22 04:08:31.842666+00:00)
gcworker stdout | 2024-04-22 04:08:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549311843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:08:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:08:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:08:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:08:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:08:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:29.931002+00:00 (in 57.996857 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:08:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:31 UTC)" (scheduled at 2024-04-22 04:08:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:08:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:08:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:08:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:42.430973+00:00 (in 9.999523 seconds)
notificationworker stdout | 2024-04-22 04:08:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:42 UTC)" (scheduled at 2024-04-22 04:08:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:08:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:08:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 32, 431853), True, datetime.datetime(2024, 4, 22, 4, 8, 32, 431853), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:08:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:08:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:08:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:08:35,395 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:08:36,682 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:08:39,269 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:08:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:08:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:10.238173+00:00 (in 29.999505 seconds)
autopruneworker stdout | 2024-04-22 04:08:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:10 UTC)" (scheduled at 2024-04-22 04:08:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:08:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758860247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:08:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:08:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:08:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:08:40,270 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:08:40,596 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:08:40,735 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:08:41,310 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:08:41,930 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:08:42,182 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:08:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:52.430973+00:00 (in 9.999526 seconds)
notificationworker stdout | 2024-04-22 04:08:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:52 UTC)" (scheduled at 2024-04-22 04:08:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:08:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:08:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 42, 431899), True, datetime.datetime(2024, 4, 22, 4, 8, 42, 431899), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:08:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:08:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:08:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:08:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:08:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:08:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:57.806778+00:00 (in 14.999448 seconds)
quotatotalworker stdout | 2024-04-22 04:08:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:57 UTC)" (scheduled at 2024-04-22 04:08:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:08:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:08:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:08:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:08:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:08:43,474 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:08:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:08:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:47.065745+00:00 (in 2.998117 seconds)
securityworker stdout | 2024-04-22 04:08:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:14 UTC)" (scheduled at 2024-04-22 04:08:44.067050+00:00)
securityworker stdout | 2024-04-22 04:08:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:08:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:08:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:08:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:08:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:08:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:08:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:08:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:08:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:08:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:08:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 3, 44, 77529), 1, 2])
securityworker stdout | 2024-04-22 04:08:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:08:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:08:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 3, 44, 77529), 1, 2])
securityworker stdout | 2024-04-22 04:08:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:08:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:08:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:08:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:08:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:08:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:08:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:08:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:08:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:08:44,429 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:08:45,632 [282] [DEBUG] [app] Starting request: urn:request:23805a4c-7d9c-4aec-9389-d3101ee93b29 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:45,632 [284] [DEBUG] [app] Starting request: urn:request:9d4fec56-8c37-4b67-9a1c-12c9f5956916 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:08:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:08:45,635 [283] [DEBUG] [app] Starting request: urn:request:a7822b07-da37-4cba-a26d-8635f462646b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:45,636 [281] [DEBUG] [app] Starting request: urn:request:8f86d7b7-f654-4db6-b429-f9c672e1a659 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:08:45,636 [283] [DEBUG] [app] Ending request: urn:request:a7822b07-da37-4cba-a26d-8635f462646b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a7822b07-da37-4cba-a26d-8635f462646b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:45,636 [281] [DEBUG] [app] Ending request: urn:request:8f86d7b7-f654-4db6-b429-f9c672e1a659 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8f86d7b7-f654-4db6-b429-f9c672e1a659', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:08:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:08:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:08:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:45,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:45,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:08:45,639 [286] [DEBUG] [app] Starting request: urn:request:fdee49c6-4a05-4e71-9a88-ac38fdaf8ffd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:08:45,639 [286] [DEBUG] [app] Ending request: urn:request:fdee49c6-4a05-4e71-9a88-ac38fdaf8ffd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fdee49c6-4a05-4e71-9a88-ac38fdaf8ffd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:08:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:45,640 [282] [DEBUG] [app] Starting request: urn:request:ff5d0e42-f473-4792-9e04-0fa7a8b95baf (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:08:45,640 [282] [DEBUG] [app] Ending request: urn:request:ff5d0e42-f473-4792-9e04-0fa7a8b95baf (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff5d0e42-f473-4792-9e04-0fa7a8b95baf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:08:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:08:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:08:45,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:08:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:08:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:08:45,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:45,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:08:45,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:08:45,661 [282] [DEBUG] [app] Ending request: urn:request:23805a4c-7d9c-4aec-9389-d3101ee93b29 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:23805a4c-7d9c-4aec-9389-d3101ee93b29', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:45,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:45,661 [284] [DEBUG] [app] Ending request: urn:request:9d4fec56-8c37-4b67-9a1c-12c9f5956916 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9d4fec56-8c37-4b67-9a1c-12c9f5956916', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:08:45,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:08:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:08:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 04:08:45,662 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:08:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
expiredappspecifictokenworker stdout | 2024-04-22 04:08:46,927 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:08:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:08:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:14.067050+00:00 (in 27.000755 seconds)
securityworker stdout | 2024-04-22 04:08:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:17 UTC)" (scheduled at 2024-04-22 04:08:47.065745+00:00)
securityworker stdout | 2024-04-22 04:08:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:08:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:08:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:08:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:08:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:08:47,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:08:47,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:17 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:08:47,121 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
namespacegcworker stdout | 2024-04-22 04:08:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:08:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:10.248668+00:00 (in 23.002298 seconds)
namespacegcworker stdout | 2024-04-22 04:08:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:47 UTC)" (scheduled at 2024-04-22 04:08:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:08:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:08:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 47, 246633), True, datetime.datetime(2024, 4, 22, 4, 8, 47, 246633), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:08:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:08:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:08:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:08:47,604 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:08:48,252 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:08:49,222 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:08:49,889 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:08:49,892 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:08:49,894 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:08:50,268 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:08:50,708 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:08:51,411 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:08:51,469 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:08:51,476 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:08:51,479 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:08:51,487 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:08:51,490 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:08:51,504 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:08:51,507 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:08:51,509 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:08:51,556 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:08:51,564 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:08:51,568 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:08:51,571 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:08:51,574 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:08:51,937 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:08:52,228 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:08:52,355 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:08:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:08:54.433683+00:00 (in 2.002237 seconds)
notificationworker stdout | 2024-04-22 04:08:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:02 UTC)" (scheduled at 2024-04-22 04:08:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:08:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:08:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 52, 431803), True, datetime.datetime(2024, 4, 22, 4, 8, 52, 431803), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:08:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:08:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:08:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:08:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:08:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:18.047111+00:00 (in 25.001072 seconds)
exportactionlogsworker stdout | 2024-04-22 04:08:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:53 UTC)" (scheduled at 2024-04-22 04:08:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:08:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:08:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 8, 53, 46475), True, datetime.datetime(2024, 4, 22, 4, 8, 53, 46475), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:08:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:08:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:08:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:53 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:08:53,191 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:08:53,194 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:08:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:08:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:02.430973+00:00 (in 7.996774 seconds)
notificationworker stdout | 2024-04-22 04:08:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:54 UTC)" (scheduled at 2024-04-22 04:08:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:08:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:08:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:09:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:08:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:08:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:12.806778+00:00 (in 14.999476 seconds)
quotatotalworker stdout | 2024-04-22 04:08:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:12 UTC)" (scheduled at 2024-04-22 04:08:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:08:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:08:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:08:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:08:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:08:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:28.130973+00:00 (in 29.999487 seconds)
buildlogsarchiver stdout | 2024-04-22 04:08:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:28 UTC)" (scheduled at 2024-04-22 04:08:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:08:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 8, 58, 131744), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:08:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:08:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:08:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:09:00,632 [286] [DEBUG] [app] Starting request: urn:request:3410448b-36e8-4866-950d-d36b306fdb84 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:00,632 [284] [DEBUG] [app] Starting request: urn:request:34ec5b05-26f0-4c2a-9cb8-a84ca4668c02 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:00,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:09:00,635 [283] [DEBUG] [app] Starting request: urn:request:0b111057-5ef0-4b72-aa44-d8b585701932 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:00,635 [280] [DEBUG] [app] Starting request: urn:request:2c3c9b48-0689-4d4b-8a3a-fb66d0d79682 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:00,635 [283] [DEBUG] [app] Ending request: urn:request:0b111057-5ef0-4b72-aa44-d8b585701932 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b111057-5ef0-4b72-aa44-d8b585701932', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:09:00,636 [280] [DEBUG] [app] Ending request: urn:request:2c3c9b48-0689-4d4b-8a3a-fb66d0d79682 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2c3c9b48-0689-4d4b-8a3a-fb66d0d79682', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:09:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:09:00,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:09:00,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:00,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:00,639 [282] [DEBUG] [app] Starting request: urn:request:6ea2949e-92d8-483c-85d9-2c2fdc86b563 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:00,639 [286] [DEBUG] [app] Starting request: urn:request:6be6faa6-4e11-4011-994d-6de15f0b4089 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:00,639 [282] [DEBUG] [app] Ending request: urn:request:6ea2949e-92d8-483c-85d9-2c2fdc86b563 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6ea2949e-92d8-483c-85d9-2c2fdc86b563', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:09:00,639 [286] [DEBUG] [app] Ending request: urn:request:6be6faa6-4e11-4011-994d-6de15f0b4089 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6be6faa6-4e11-4011-994d-6de15f0b4089', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:09:00,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:09:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:00,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:00,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:00,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:00,661 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:00,665 [286] [DEBUG] [app] Ending request: urn:request:3410448b-36e8-4866-950d-d36b306fdb84 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3410448b-36e8-4866-950d-d36b306fdb84', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:00,665 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:00,665 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 04:09:00,673 [284] [DEBUG] [app] Ending request: urn:request:34ec5b05-26f0-4c2a-9cb8-a84ca4668c02 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:34ec5b05-26f0-4c2a-9cb8-a84ca4668c02', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:00,673 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:00,673 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.042 118 0.042)
gcworker stdout | 2024-04-22 04:09:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:09:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:31.842666+00:00 (in 29.999461 seconds)
gcworker stdout | 2024-04-22 04:09:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:31 UTC)" (scheduled at 2024-04-22 04:09:01.842666+00:00)
gcworker stdout | 2024-04-22 04:09:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549341843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:09:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:09:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:09:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:09:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:09:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:09.033206+00:00 (in 6.997021 seconds)
repositorygcworker stdout | 2024-04-22 04:09:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:02 UTC)" (scheduled at 2024-04-22 04:09:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:09:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:09:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:09:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:12.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 04:09:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:12 UTC)" (scheduled at 2024-04-22 04:09:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:09:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:09:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 2, 431875), True, datetime.datetime(2024, 4, 22, 4, 9, 2, 431875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:09:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:09:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:09:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:09:05,422 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:09:06,718 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:09:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:09:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:02.035613+00:00 (in 53.001915 seconds)
repositorygcworker stdout | 2024-04-22 04:09:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:09 UTC)" (scheduled at 2024-04-22 04:09:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:09:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:09:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 9, 34236), True, datetime.datetime(2024, 4, 22, 4, 9, 9, 34236), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:09:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:09:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:09:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:09:09,304 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:09:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:09:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:40.238173+00:00 (in 29.999534 seconds)
autopruneworker stdout | 2024-04-22 04:09:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:40 UTC)" (scheduled at 2024-04-22 04:09:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:09:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758890247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:09:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:09:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:47.245913+00:00 (in 36.996792 seconds)
namespacegcworker stdout | 2024-04-22 04:09:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:10 UTC)" (scheduled at 2024-04-22 04:09:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:09:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:09:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:09:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:09:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:09:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:09:10,286 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:09:10,632 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:09:10,766 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:09:11,317 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:09:11,949 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:09:12,220 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:09:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:22.430973+00:00 (in 9.999468 seconds)
notificationworker stdout | 2024-04-22 04:09:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:22 UTC)" (scheduled at 2024-04-22 04:09:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:09:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:09:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 12, 431790), True, datetime.datetime(2024, 4, 22, 4, 9, 12, 431790), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:09:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:09:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:09:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:09:12,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:09:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:27.806778+00:00 (in 14.999509 seconds)
quotatotalworker stdout | 2024-04-22 04:09:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:27 UTC)" (scheduled at 2024-04-22 04:09:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:09:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:09:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:09:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:09:13,511 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:09:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:09:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:17.065745+00:00 (in 2.998145 seconds)
securityworker stdout | 2024-04-22 04:09:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:44 UTC)" (scheduled at 2024-04-22 04:09:14.067050+00:00)
securityworker stdout | 2024-04-22 04:09:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:09:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:09:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:09:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:09:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:09:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:09:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:09:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:09:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:09:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:09:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 4, 14, 78709), 1, 2])
securityworker stdout | 2024-04-22 04:09:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:09:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:09:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 4, 14, 78709), 1, 2])
securityworker stdout | 2024-04-22 04:09:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:09:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:09:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:09:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:09:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:09:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:09:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:09:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:09:14,461 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:09:15,632 [282] [DEBUG] [app] Starting request: urn:request:387136f4-d1ef-4b01-a578-b3211601b8aa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:15,632 [284] [DEBUG] [app] Starting request: urn:request:9153563a-b19e-4115-ac7c-e951b7d8a2b1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:09:15,635 [283] [DEBUG] [app] Starting request: urn:request:5cfcd1b6-bbbf-4afa-8e58-6e462adc4989 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:15,635 [275] [DEBUG] [app] Starting request: urn:request:620c96ec-2f21-4b5f-a0a2-c032aa138738 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:15,635 [283] [DEBUG] [app] Ending request: urn:request:5cfcd1b6-bbbf-4afa-8e58-6e462adc4989 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cfcd1b6-bbbf-4afa-8e58-6e462adc4989', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:09:15,635 [275] [DEBUG] [app] Ending request: urn:request:620c96ec-2f21-4b5f-a0a2-c032aa138738 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:620c96ec-2f21-4b5f-a0a2-c032aa138738', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:09:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:09:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:15,638 [282] [DEBUG] [app] Starting request: urn:request:2fb9bfd3-2081-41a1-bbad-4c6a103ba249 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:15,638 [282] [DEBUG] [app] Ending request: urn:request:2fb9bfd3-2081-41a1-bbad-4c6a103ba249 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2fb9bfd3-2081-41a1-bbad-4c6a103ba249', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:09:15,638 [286] [DEBUG] [app] Starting request: urn:request:c26f7d8f-6514-445a-9406-2fdc444f5f8d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:09:15,639 [286] [DEBUG] [app] Ending request: urn:request:c26f7d8f-6514-445a-9406-2fdc444f5f8d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c26f7d8f-6514-445a-9406-2fdc444f5f8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:09:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:15,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:09:15,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:15,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:15,659 [282] [DEBUG] [app] Ending request: urn:request:387136f4-d1ef-4b01-a578-b3211601b8aa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:387136f4-d1ef-4b01-a578-b3211601b8aa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:15,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:15,659 [284] [DEBUG] [app] Ending request: urn:request:9153563a-b19e-4115-ac7c-e951b7d8a2b1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9153563a-b19e-4115-ac7c-e951b7d8a2b1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:15,659 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:15,659 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:09:15,659 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
expiredappspecifictokenworker stdout | 2024-04-22 04:09:16,963 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:09:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:09:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:44.067050+00:00 (in 27.000764 seconds)
securityworker stdout | 2024-04-22 04:09:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:47 UTC)" (scheduled at 2024-04-22 04:09:17.065745+00:00)
securityworker stdout | 2024-04-22 04:09:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:09:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:09:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:09:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:09:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:09:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:09:17,085 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:09:17,141 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:09:17,632 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:09:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:09:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:53.045554+00:00 (in 34.997939 seconds)
exportactionlogsworker stdout | 2024-04-22 04:09:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:18 UTC)" (scheduled at 2024-04-22 04:09:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:09:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:09:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:09:18,274 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:09:19,253 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:09:19,897 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:09:19,900 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:09:19,902 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:09:20,305 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:09:20,745 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:09:21,420 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:09:21,479 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:09:21,485 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:09:21,487 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:09:21,495 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:09:21,498 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:09:21,512 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:09:21,515 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:09:21,518 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:09:21,562 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:09:21,571 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:09:21,575 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:09:21,578 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:09:21,581 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:09:21,974 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:09:22,249 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:09:22,376 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:09:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:32.430973+00:00 (in 9.999469 seconds)
notificationworker stdout | 2024-04-22 04:09:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:32 UTC)" (scheduled at 2024-04-22 04:09:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:09:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:09:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 22, 431938), True, datetime.datetime(2024, 4, 22, 4, 9, 22, 431938), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:09:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:09:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:09:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:09:23,209 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:09:23,228 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:09:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:09:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:24.738365+00:00 (in 59.999497 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:09:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:24 UTC)" (scheduled at 2024-04-22 04:09:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:09:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:09:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:09:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:09:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:09:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:42.806778+00:00 (in 14.999473 seconds)
quotatotalworker stdout | 2024-04-22 04:09:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:42 UTC)" (scheduled at 2024-04-22 04:09:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:09:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:09:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:09:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:09:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:09:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:58.130973+00:00 (in 29.999472 seconds)
buildlogsarchiver stdout | 2024-04-22 04:09:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:58 UTC)" (scheduled at 2024-04-22 04:09:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:09:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 9, 28, 131754), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:09:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:09:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:09:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:09:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:31.933721+00:00 (in 2.002214 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:29 UTC)" (scheduled at 2024-04-22 04:09:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 29, 931782), True, datetime.datetime(2024, 4, 22, 4, 9, 29, 931782), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:09:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:09:30,632 [285] [DEBUG] [app] Starting request: urn:request:de1afda4-2e34-4029-b20c-5a31d61b966f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:30,633 [284] [DEBUG] [app] Starting request: urn:request:c2274334-7097-4cdb-9129-e7d3123b62ce (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:09:30,636 [283] [DEBUG] [app] Starting request: urn:request:104348fe-e617-4fd6-89bd-9e17ef4f81df (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:30,636 [280] [DEBUG] [app] Starting request: urn:request:4aef4f84-c346-46a8-9c60-589b9c0aef83 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:30,636 [283] [DEBUG] [app] Ending request: urn:request:104348fe-e617-4fd6-89bd-9e17ef4f81df (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:104348fe-e617-4fd6-89bd-9e17ef4f81df', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:09:30,636 [280] [DEBUG] [app] Ending request: urn:request:4aef4f84-c346-46a8-9c60-589b9c0aef83 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4aef4f84-c346-46a8-9c60-589b9c0aef83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:09:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:09:30,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:30,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:30,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:09:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:30,640 [282] [DEBUG] [app] Starting request: urn:request:a46db7dc-a19c-4d91-ac53-88cad94b9569 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:30,640 [286] [DEBUG] [app] Starting request: urn:request:6826b02b-82fc-4b02-a267-fed344135728 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:30,640 [286] [DEBUG] [app] Ending request: urn:request:6826b02b-82fc-4b02-a267-fed344135728 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6826b02b-82fc-4b02-a267-fed344135728', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:09:30,640 [282] [DEBUG] [app] Ending request: urn:request:a46db7dc-a19c-4d91-ac53-88cad94b9569 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a46db7dc-a19c-4d91-ac53-88cad94b9569', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:09:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:30,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:30,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:30,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:30,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:30,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:30,661 [285] [DEBUG] [app] Ending request: urn:request:de1afda4-2e34-4029-b20c-5a31d61b966f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de1afda4-2e34-4029-b20c-5a31d61b966f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:30,661 [284] [DEBUG] [app] Ending request: urn:request:c2274334-7097-4cdb-9129-e7d3123b62ce (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c2274334-7097-4cdb-9129-e7d3123b62ce', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:30,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:30,662 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:09:30,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:09:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:09:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:01.842666+00:00 (in 29.999478 seconds)
gcworker stdout | 2024-04-22 04:09:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:01 UTC)" (scheduled at 2024-04-22 04:09:31.842666+00:00)
gcworker stdout | 2024-04-22 04:09:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549371843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:09:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:09:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:09:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:09:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:09:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:29.931002+00:00 (in 57.996838 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:09:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:31 UTC)" (scheduled at 2024-04-22 04:09:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:09:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:09:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:09:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:42.430973+00:00 (in 9.999547 seconds)
notificationworker stdout | 2024-04-22 04:09:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:42 UTC)" (scheduled at 2024-04-22 04:09:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:09:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:09:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 32, 431838), True, datetime.datetime(2024, 4, 22, 4, 9, 32, 431838), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:09:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:09:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:09:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:09:35,458 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:09:36,736 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:09:39,340 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:09:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:09:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:10.238173+00:00 (in 29.999484 seconds)
autopruneworker stdout | 2024-04-22 04:09:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:10 UTC)" (scheduled at 2024-04-22 04:09:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:09:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758920247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:09:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:09:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:09:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:09:40,322 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:09:40,639 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:09:40,802 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:09:41,353 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:09:41,985 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:09:42,236 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:09:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:52.430973+00:00 (in 9.999509 seconds)
notificationworker stdout | 2024-04-22 04:09:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:52 UTC)" (scheduled at 2024-04-22 04:09:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:09:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:09:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 42, 431885), True, datetime.datetime(2024, 4, 22, 4, 9, 42, 431885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:09:42,453 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:09:42,453 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:09:42,454 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:09:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:09:42,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:09:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:57.806778+00:00 (in 14.999497 seconds)
quotatotalworker stdout | 2024-04-22 04:09:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:57 UTC)" (scheduled at 2024-04-22 04:09:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:09:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:09:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:09:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:09:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:09:43,519 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:09:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:09:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:47.065745+00:00 (in 2.998207 seconds)
securityworker stdout | 2024-04-22 04:09:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:14 UTC)" (scheduled at 2024-04-22 04:09:44.067050+00:00)
securityworker stdout | 2024-04-22 04:09:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:09:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:09:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:09:44,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:09:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:09:44,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:09:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:09:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:09:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:09:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:09:44,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 4, 44, 76889), 1, 2])
securityworker stdout | 2024-04-22 04:09:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:09:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:09:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 4, 44, 76889), 1, 2])
securityworker stdout | 2024-04-22 04:09:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:09:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:09:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:09:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:09:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:09:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:09:44,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:09:44,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:09:44,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:09:44,485 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:09:45,632 [282] [DEBUG] [app] Starting request: urn:request:f0a3560e-fa49-48f7-abb6-74b96afe24ec (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:45,632 [284] [DEBUG] [app] Starting request: urn:request:ef463fb6-5648-4fd1-afa8-9883e18e5731 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:09:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:09:45,635 [275] [DEBUG] [app] Starting request: urn:request:bd1b7f64-d200-49ff-840a-d597198ab795 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:45,635 [283] [DEBUG] [app] Starting request: urn:request:c379e6ba-d06b-47e4-8e24-b0e60877614d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:09:45,636 [283] [DEBUG] [app] Ending request: urn:request:c379e6ba-d06b-47e4-8e24-b0e60877614d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c379e6ba-d06b-47e4-8e24-b0e60877614d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:09:45,636 [275] [DEBUG] [app] Ending request: urn:request:bd1b7f64-d200-49ff-840a-d597198ab795 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd1b7f64-d200-49ff-840a-d597198ab795', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:09:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:09:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:45,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:09:45,639 [285] [DEBUG] [app] Starting request: urn:request:b3eae62f-fae6-404a-bfb0-f356fd6b36a8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:45,639 [282] [DEBUG] [app] Starting request: urn:request:57f9e898-f6e1-47af-85db-0ecf4d28b33d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:09:45,639 [285] [DEBUG] [app] Ending request: urn:request:b3eae62f-fae6-404a-bfb0-f356fd6b36a8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b3eae62f-fae6-404a-bfb0-f356fd6b36a8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:09:45,639 [282] [DEBUG] [app] Ending request: urn:request:57f9e898-f6e1-47af-85db-0ecf4d28b33d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:57f9e898-f6e1-47af-85db-0ecf4d28b33d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:09:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:09:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:09:45,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:09:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:09:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:09:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:45,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:09:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:45,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:09:45,660 [284] [DEBUG] [app] Ending request: urn:request:ef463fb6-5648-4fd1-afa8-9883e18e5731 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ef463fb6-5648-4fd1-afa8-9883e18e5731', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:45,660 [282] [DEBUG] [app] Ending request: urn:request:f0a3560e-fa49-48f7-abb6-74b96afe24ec (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0a3560e-fa49-48f7-abb6-74b96afe24ec', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:09:45,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:09:45,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:09:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:09:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:09:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
expiredappspecifictokenworker stdout | 2024-04-22 04:09:47,000 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:09:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:09:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:14.067050+00:00 (in 27.000791 seconds)
securityworker stdout | 2024-04-22 04:09:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:17 UTC)" (scheduled at 2024-04-22 04:09:47.065745+00:00)
securityworker stdout | 2024-04-22 04:09:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:09:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:09:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:09:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:09:47,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:09:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:09:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:17 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:09:47,167 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
namespacegcworker stdout | 2024-04-22 04:09:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:09:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:10.248668+00:00 (in 23.002257 seconds)
namespacegcworker stdout | 2024-04-22 04:09:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:47 UTC)" (scheduled at 2024-04-22 04:09:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:09:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:09:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 47, 246696), True, datetime.datetime(2024, 4, 22, 4, 9, 47, 246696), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:09:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:09:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:09:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:09:47,669 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:09:48,305 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:09:49,261 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:09:49,905 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:09:49,908 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:09:49,910 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:09:50,342 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:09:50,781 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:09:51,428 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:09:51,488 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:09:51,492 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:09:51,498 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:09:51,502 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:09:51,505 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:09:51,520 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:09:51,523 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:09:51,528 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:09:51,569 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:09:51,578 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:09:51,582 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:09:51,587 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:09:51,589 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:09:52,010 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:09:52,269 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:09:52,386 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:09:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:09:54.433683+00:00 (in 2.002217 seconds)
notificationworker stdout | 2024-04-22 04:09:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:02 UTC)" (scheduled at 2024-04-22 04:09:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:09:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:09:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 52, 431879), True, datetime.datetime(2024, 4, 22, 4, 9, 52, 431879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:09:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:09:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:09:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:09:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:09:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:18.047111+00:00 (in 25.001072 seconds)
exportactionlogsworker stdout | 2024-04-22 04:09:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:53 UTC)" (scheduled at 2024-04-22 04:09:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:09:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:09:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 9, 53, 46459), True, datetime.datetime(2024, 4, 22, 4, 9, 53, 46459), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:09:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:09:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:09:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:09:53,229 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:09:53,242 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:09:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:09:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:02.430973+00:00 (in 7.996800 seconds)
notificationworker stdout | 2024-04-22 04:09:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:54 UTC)" (scheduled at 2024-04-22 04:09:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:09:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:09:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:10:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:09:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:09:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:12.806778+00:00 (in 14.999537 seconds)
quotatotalworker stdout | 2024-04-22 04:09:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:12 UTC)" (scheduled at 2024-04-22 04:09:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:09:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:09:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:09:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:09:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:09:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:28.130973+00:00 (in 29.999500 seconds)
buildlogsarchiver stdout | 2024-04-22 04:09:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:28 UTC)" (scheduled at 2024-04-22 04:09:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:09:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 9, 58, 131706), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:09:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:09:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:09:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:10:00,633 [285] [DEBUG] [app] Starting request: urn:request:deb476e3-bd21-44fe-82cc-c9ec62d79268 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:00,633 [282] [DEBUG] [app] Starting request: urn:request:c47750c2-998d-4632-ba42-20d4447e2aae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:10:00,636 [283] [DEBUG] [app] Starting request: urn:request:7da1b4ac-e2bc-451f-b033-60fa38b19ef6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:00,636 [275] [DEBUG] [app] Starting request: urn:request:439a7563-189b-4d37-87a9-69fb687126ec (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:00,636 [275] [DEBUG] [app] Ending request: urn:request:439a7563-189b-4d37-87a9-69fb687126ec (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:439a7563-189b-4d37-87a9-69fb687126ec', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:10:00,636 [283] [DEBUG] [app] Ending request: urn:request:7da1b4ac-e2bc-451f-b033-60fa38b19ef6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7da1b4ac-e2bc-451f-b033-60fa38b19ef6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:10:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:10:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:00,639 [285] [DEBUG] [app] Starting request: urn:request:a801e3a1-a1d8-4a82-be8e-7f0b2d8f00e3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:00,639 [285] [DEBUG] [app] Ending request: urn:request:a801e3a1-a1d8-4a82-be8e-7f0b2d8f00e3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a801e3a1-a1d8-4a82-be8e-7f0b2d8f00e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:10:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:00,641 [284] [DEBUG] [app] Starting request: urn:request:0a1258df-17b8-471c-b33f-7b14c0c6bb08 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:00,641 [284] [DEBUG] [app] Ending request: urn:request:0a1258df-17b8-471c-b33f-7b14c0c6bb08 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0a1258df-17b8-471c-b33f-7b14c0c6bb08', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:00,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:10:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:00,660 [285] [DEBUG] [app] Ending request: urn:request:deb476e3-bd21-44fe-82cc-c9ec62d79268 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:deb476e3-bd21-44fe-82cc-c9ec62d79268', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:00,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:00,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:10:00,661 [282] [DEBUG] [app] Ending request: urn:request:c47750c2-998d-4632-ba42-20d4447e2aae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c47750c2-998d-4632-ba42-20d4447e2aae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:10:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:10:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:31.842666+00:00 (in 29.999529 seconds)
gcworker stdout | 2024-04-22 04:10:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:31 UTC)" (scheduled at 2024-04-22 04:10:01.842666+00:00)
gcworker stdout | 2024-04-22 04:10:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549401843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:10:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:10:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:10:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:10:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:10:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:09.033206+00:00 (in 6.997082 seconds)
repositorygcworker stdout | 2024-04-22 04:10:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:02 UTC)" (scheduled at 2024-04-22 04:10:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:10:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:10:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:10:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:12.430973+00:00 (in 9.999490 seconds)
notificationworker stdout | 2024-04-22 04:10:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:12 UTC)" (scheduled at 2024-04-22 04:10:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:10:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:10:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 2, 431879), True, datetime.datetime(2024, 4, 22, 4, 10, 2, 431879), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:10:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:10:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:10:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:10:05,496 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:10:06,768 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:10:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:10:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:02.035613+00:00 (in 53.001896 seconds)
repositorygcworker stdout | 2024-04-22 04:10:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:09 UTC)" (scheduled at 2024-04-22 04:10:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:10:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:10:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 9, 34195), True, datetime.datetime(2024, 4, 22, 4, 10, 9, 34195), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:10:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:10:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:10:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:10:09,367 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:10:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:10:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:40.238173+00:00 (in 29.999458 seconds)
autopruneworker stdout | 2024-04-22 04:10:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:40 UTC)" (scheduled at 2024-04-22 04:10:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:10:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758950247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:10:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:10:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:47.245913+00:00 (in 36.996791 seconds)
namespacegcworker stdout | 2024-04-22 04:10:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:10 UTC)" (scheduled at 2024-04-22 04:10:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:10:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:10:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:10:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:10:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:10:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:10:10,358 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:10:10,648 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:10:10,839 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:10:11,384 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:10:12,022 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:10:12,271 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:10:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:22.430973+00:00 (in 9.999555 seconds)
notificationworker stdout | 2024-04-22 04:10:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:22 UTC)" (scheduled at 2024-04-22 04:10:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:10:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:10:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 12, 431871), True, datetime.datetime(2024, 4, 22, 4, 10, 12, 431871), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:10:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:10:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:10:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:10:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:10:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:27.806778+00:00 (in 14.999459 seconds)
quotatotalworker stdout | 2024-04-22 04:10:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:27 UTC)" (scheduled at 2024-04-22 04:10:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:10:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:10:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:10:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:10:13,547 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:10:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:10:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:17.065745+00:00 (in 2.998156 seconds)
securityworker stdout | 2024-04-22 04:10:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:44 UTC)" (scheduled at 2024-04-22 04:10:14.067050+00:00)
securityworker stdout | 2024-04-22 04:10:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:10:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:10:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:10:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:10:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:10:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:10:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:10:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:10:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:10:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:10:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 5, 14, 77590), 1, 2])
securityworker stdout | 2024-04-22 04:10:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:10:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:10:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 5, 14, 77590), 1, 2])
securityworker stdout | 2024-04-22 04:10:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:10:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:10:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:10:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:10:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:10:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:10:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:10:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:10:14,520 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:10:15,632 [285] [DEBUG] [app] Starting request: urn:request:426c272a-e1c3-4e4f-b274-3bbac0bbd2e4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:15,632 [286] [DEBUG] [app] Starting request: urn:request:b2dd8290-2572-41ee-a7f0-28b8fee4ecda (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:10:15,635 [283] [DEBUG] [app] Starting request: urn:request:1669b247-1a8c-437f-abb6-ad74c89598d5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:15,635 [279] [DEBUG] [app] Starting request: urn:request:8c395efe-2f4c-415a-9a85-7f2ba84fa85a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:15,635 [283] [DEBUG] [app] Ending request: urn:request:1669b247-1a8c-437f-abb6-ad74c89598d5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1669b247-1a8c-437f-abb6-ad74c89598d5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:10:15,635 [279] [DEBUG] [app] Ending request: urn:request:8c395efe-2f4c-415a-9a85-7f2ba84fa85a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c395efe-2f4c-415a-9a85-7f2ba84fa85a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:10:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:10:15,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:15,638 [282] [DEBUG] [app] Starting request: urn:request:c5eb1f55-8edd-41cb-87f3-285ba139fdfe (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:15,638 [285] [DEBUG] [app] Starting request: urn:request:32b3099f-85c3-4936-8500-43b6269eeea8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:15,639 [285] [DEBUG] [app] Ending request: urn:request:32b3099f-85c3-4936-8500-43b6269eeea8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32b3099f-85c3-4936-8500-43b6269eeea8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:15,639 [282] [DEBUG] [app] Ending request: urn:request:c5eb1f55-8edd-41cb-87f3-285ba139fdfe (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c5eb1f55-8edd-41cb-87f3-285ba139fdfe', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:10:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:15,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:15,659 [286] [DEBUG] [app] Ending request: urn:request:b2dd8290-2572-41ee-a7f0-28b8fee4ecda (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b2dd8290-2572-41ee-a7f0-28b8fee4ecda', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:15,660 [285] [DEBUG] [app] Ending request: urn:request:426c272a-e1c3-4e4f-b274-3bbac0bbd2e4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:426c272a-e1c3-4e4f-b274-3bbac0bbd2e4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:10:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
expiredappspecifictokenworker stdout | 2024-04-22 04:10:17,036 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:10:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:10:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:44.067050+00:00 (in 27.000782 seconds)
securityworker stdout | 2024-04-22 04:10:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:47 UTC)" (scheduled at 2024-04-22 04:10:17.065745+00:00)
securityworker stdout | 2024-04-22 04:10:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:10:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:10:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:10:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:10:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:10:17,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:10:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:10:17,184 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:10:17,704 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:10:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:10:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:53.045554+00:00 (in 34.997929 seconds)
exportactionlogsworker stdout | 2024-04-22 04:10:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:18 UTC)" (scheduled at 2024-04-22 04:10:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:10:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:10:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:10:18,321 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:10:19,278 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:10:19,914 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:10:19,917 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:10:19,919 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:10:20,358 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:10:20,817 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:10:21,440 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:10:21,497 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:10:21,505 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:10:21,508 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:10:21,510 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:10:21,515 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:10:21,530 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:10:21,534 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:10:21,538 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:10:21,576 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:10:21,584 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:10:21,589 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:10:21,595 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:10:21,597 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:10:22,029 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:10:22,304 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:10:22,422 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:10:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:32.430973+00:00 (in 9.999552 seconds)
notificationworker stdout | 2024-04-22 04:10:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:32 UTC)" (scheduled at 2024-04-22 04:10:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:10:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:10:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 22, 431862), True, datetime.datetime(2024, 4, 22, 4, 10, 22, 431862), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:10:22,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:10:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:10:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:10:23,245 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:10:23,263 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:10:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:10:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:24.738365+00:00 (in 59.999477 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:10:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:24 UTC)" (scheduled at 2024-04-22 04:10:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:10:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:10:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:10:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:10:27,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:10:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:42.806778+00:00 (in 14.999479 seconds)
quotatotalworker stdout | 2024-04-22 04:10:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:42 UTC)" (scheduled at 2024-04-22 04:10:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:10:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:10:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:10:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:10:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:10:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:58.130973+00:00 (in 29.999458 seconds)
buildlogsarchiver stdout | 2024-04-22 04:10:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:58 UTC)" (scheduled at 2024-04-22 04:10:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:10:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 10, 28, 131786), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:10:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:10:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:10:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:10:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:31.933721+00:00 (in 2.002270 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:29 UTC)" (scheduled at 2024-04-22 04:10:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 29, 931708), True, datetime.datetime(2024, 4, 22, 4, 10, 29, 931708), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:10:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:10:30,632 [282] [DEBUG] [app] Starting request: urn:request:9bef8c89-95bb-4f3e-8414-6d5ff7fa7d6f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:30,632 [286] [DEBUG] [app] Starting request: urn:request:95503c4e-bd03-4ea6-ad89-76da1738d311 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:10:30,635 [275] [DEBUG] [app] Starting request: urn:request:938d051d-c4b3-4b1b-8401-e005b8ccdf15 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:30,635 [276] [DEBUG] [app] Starting request: urn:request:0b43b6de-a8dc-4b85-a593-0fb6ef36e11e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:30,636 [275] [DEBUG] [app] Ending request: urn:request:938d051d-c4b3-4b1b-8401-e005b8ccdf15 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:938d051d-c4b3-4b1b-8401-e005b8ccdf15', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:10:30,636 [276] [DEBUG] [app] Ending request: urn:request:0b43b6de-a8dc-4b85-a593-0fb6ef36e11e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0b43b6de-a8dc-4b85-a593-0fb6ef36e11e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:10:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:10:30,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:10:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:30,639 [282] [DEBUG] [app] Starting request: urn:request:a5c616bf-164c-4025-ab50-2991ee6f8082 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:30,639 [282] [DEBUG] [app] Ending request: urn:request:a5c616bf-164c-4025-ab50-2991ee6f8082 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a5c616bf-164c-4025-ab50-2991ee6f8082', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:10:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:30,640 [284] [DEBUG] [app] Starting request: urn:request:c93c7daa-8a35-46b8-8538-5a9bae0eee63 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:30,640 [284] [DEBUG] [app] Ending request: urn:request:c93c7daa-8a35-46b8-8538-5a9bae0eee63 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c93c7daa-8a35-46b8-8538-5a9bae0eee63', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:10:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:30,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:30,659 [282] [DEBUG] [app] Ending request: urn:request:9bef8c89-95bb-4f3e-8414-6d5ff7fa7d6f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9bef8c89-95bb-4f3e-8414-6d5ff7fa7d6f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:30,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:30,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:10:30,660 [286] [DEBUG] [app] Ending request: urn:request:95503c4e-bd03-4ea6-ad89-76da1738d311 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95503c4e-bd03-4ea6-ad89-76da1738d311', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:10:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:10:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:01.842666+00:00 (in 29.999473 seconds)
gcworker stdout | 2024-04-22 04:10:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:01 UTC)" (scheduled at 2024-04-22 04:10:31.842666+00:00)
gcworker stdout | 2024-04-22 04:10:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549431843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:10:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:10:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:10:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:10:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:10:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:29.931002+00:00 (in 57.996788 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:10:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:31 UTC)" (scheduled at 2024-04-22 04:10:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:10:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:10:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:10:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:42.430973+00:00 (in 9.999543 seconds)
notificationworker stdout | 2024-04-22 04:10:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:42 UTC)" (scheduled at 2024-04-22 04:10:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:10:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:10:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 32, 431708), True, datetime.datetime(2024, 4, 22, 4, 10, 32, 431708), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:10:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:10:32,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:10:32,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:10:35,507 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:10:36,775 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:10:39,375 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:10:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:10:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:10.238173+00:00 (in 29.999464 seconds)
autopruneworker stdout | 2024-04-22 04:10:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:10 UTC)" (scheduled at 2024-04-22 04:10:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:10:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713758980247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:10:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:10:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:10:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:10:40,394 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:10:40,684 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:10:40,875 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:10:41,418 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:10:42,058 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:10:42,305 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:10:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:52.430973+00:00 (in 9.999550 seconds)
notificationworker stdout | 2024-04-22 04:10:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:52 UTC)" (scheduled at 2024-04-22 04:10:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:10:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:10:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 42, 431875), True, datetime.datetime(2024, 4, 22, 4, 10, 42, 431875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:10:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:10:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:10:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:10:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:10:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:10:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:57.806778+00:00 (in 14.999453 seconds)
quotatotalworker stdout | 2024-04-22 04:10:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:57 UTC)" (scheduled at 2024-04-22 04:10:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:10:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:10:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:10:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:10:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:10:43,565 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:10:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:10:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:47.065745+00:00 (in 2.998146 seconds)
securityworker stdout | 2024-04-22 04:10:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:14 UTC)" (scheduled at 2024-04-22 04:10:44.067050+00:00)
securityworker stdout | 2024-04-22 04:10:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:10:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:10:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:10:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:10:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:10:44,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:10:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:10:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:10:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:10:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:10:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 5, 44, 78308), 1, 2])
securityworker stdout | 2024-04-22 04:10:44,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:10:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:10:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 5, 44, 78308), 1, 2])
securityworker stdout | 2024-04-22 04:10:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:10:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:10:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:10:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:10:44,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:10:44,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:10:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:10:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:10:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:10:44,556 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:10:45,631 [286] [DEBUG] [app] Starting request: urn:request:ecd4f6c9-c7e5-4b49-83d6-a34f9203434e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:45,631 [284] [DEBUG] [app] Starting request: urn:request:317879e9-5fce-4943-ba94-f45db8e0dcb4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:10:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:10:45,635 [275] [DEBUG] [app] Starting request: urn:request:0bbee458-6655-465c-ba31-cf2c0f72ad17 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:45,635 [275] [DEBUG] [app] Ending request: urn:request:0bbee458-6655-465c-ba31-cf2c0f72ad17 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0bbee458-6655-465c-ba31-cf2c0f72ad17', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:10:45,635 [283] [DEBUG] [app] Starting request: urn:request:7eea4157-14f5-4399-9aad-9fd67fb26176 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:10:45,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:10:45,635 [283] [DEBUG] [app] Ending request: urn:request:7eea4157-14f5-4399-9aad-9fd67fb26176 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:7eea4157-14f5-4399-9aad-9fd67fb26176', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:10:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 04:10:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:10:45,639 [282] [DEBUG] [app] Starting request: urn:request:86fdcc42-cb63-4855-b31c-94b83791bfe9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:10:45,639 [282] [DEBUG] [app] Ending request: urn:request:86fdcc42-cb63-4855-b31c-94b83791bfe9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:86fdcc42-cb63-4855-b31c-94b83791bfe9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:45,639 [285] [DEBUG] [app] Starting request: urn:request:9cb5f4ad-b56e-42f7-b46e-f80288464b4e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:10:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:45,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:45,640 [285] [DEBUG] [app] Ending request: urn:request:9cb5f4ad-b56e-42f7-b46e-f80288464b4e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9cb5f4ad-b56e-42f7-b46e-f80288464b4e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:10:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:10:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:10:45,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:10:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:10:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:10:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:10:45,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:10:45,660 [284] [DEBUG] [app] Ending request: urn:request:317879e9-5fce-4943-ba94-f45db8e0dcb4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:317879e9-5fce-4943-ba94-f45db8e0dcb4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:45,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:10:45,660 [286] [DEBUG] [app] Ending request: urn:request:ecd4f6c9-c7e5-4b49-83d6-a34f9203434e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ecd4f6c9-c7e5-4b49-83d6-a34f9203434e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:10:45,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:10:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:10:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:10:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:10:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:10:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:10:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:14.067050+00:00 (in 27.000732 seconds)
securityworker stdout | 2024-04-22 04:10:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:17 UTC)" (scheduled at 2024-04-22 04:10:47.065745+00:00)
securityworker stdout | 2024-04-22 04:10:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:10:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
expiredappspecifictokenworker stdout | 2024-04-22 04:10:47,067 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:10:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:10:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:10:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:10:47,085 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:10:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:17 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:10:47,221 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
namespacegcworker stdout | 2024-04-22 04:10:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:10:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:10.248668+00:00 (in 23.002318 seconds)
namespacegcworker stdout | 2024-04-22 04:10:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:47 UTC)" (scheduled at 2024-04-22 04:10:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:10:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:10:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 47, 246603), True, datetime.datetime(2024, 4, 22, 4, 10, 47, 246603), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:10:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:10:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:10:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:47 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:10:47,741 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:10:48,358 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:10:49,303 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:10:49,923 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:10:49,926 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:10:49,929 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:10:50,395 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:10:50,849 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:10:51,451 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:10:51,508 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:10:51,515 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:10:51,517 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:10:51,520 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:10:51,526 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:10:51,540 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:10:51,544 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:10:51,547 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:10:51,582 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:10:51,591 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:10:51,596 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:10:51,601 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:10:51,604 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:10:52,065 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:10:52,340 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:10:52,428 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:10:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:10:54.433683+00:00 (in 2.002282 seconds)
notificationworker stdout | 2024-04-22 04:10:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:02 UTC)" (scheduled at 2024-04-22 04:10:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:10:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:10:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 52, 431750), True, datetime.datetime(2024, 4, 22, 4, 10, 52, 431750), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:10:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:10:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:10:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:02 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:10:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:10:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:18.047111+00:00 (in 25.001061 seconds)
exportactionlogsworker stdout | 2024-04-22 04:10:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:53 UTC)" (scheduled at 2024-04-22 04:10:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:10:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:10:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 10, 53, 46474), True, datetime.datetime(2024, 4, 22, 4, 10, 53, 46474), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:10:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:10:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:10:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:10:53,252 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:10:53,298 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:10:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:10:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:02.430973+00:00 (in 7.996751 seconds)
notificationworker stdout | 2024-04-22 04:10:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:54 UTC)" (scheduled at 2024-04-22 04:10:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:10:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:10:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:11:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:10:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:10:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:12.806778+00:00 (in 14.999496 seconds)
quotatotalworker stdout | 2024-04-22 04:10:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:12 UTC)" (scheduled at 2024-04-22 04:10:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:10:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:10:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:10:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:10:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:10:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:28.130973+00:00 (in 29.999455 seconds)
buildlogsarchiver stdout | 2024-04-22 04:10:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:28 UTC)" (scheduled at 2024-04-22 04:10:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:10:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 10, 58, 131767), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:10:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:10:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:10:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:11:00,632 [286] [DEBUG] [app] Starting request: urn:request:5efa1270-1ed3-42ba-ad29-1a33d75c1524 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:00,633 [282] [DEBUG] [app] Starting request: urn:request:044cb297-54ba-4b3f-beb5-d18a19980e19 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:11:00,635 [283] [DEBUG] [app] Starting request: urn:request:f0dd85fa-d104-4147-a71a-ed69a7b01602 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:00,636 [283] [DEBUG] [app] Ending request: urn:request:f0dd85fa-d104-4147-a71a-ed69a7b01602 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f0dd85fa-d104-4147-a71a-ed69a7b01602', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:11:00,636 [280] [DEBUG] [app] Starting request: urn:request:151b791f-f8aa-41d9-9ff9-a8ed89b77ba6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:11:00,636 [280] [DEBUG] [app] Ending request: urn:request:151b791f-f8aa-41d9-9ff9-a8ed89b77ba6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:151b791f-f8aa-41d9-9ff9-a8ed89b77ba6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:11:00,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:00,639 [286] [DEBUG] [app] Starting request: urn:request:0147066a-c96d-470e-a92b-c230cac8630b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:00,639 [286] [DEBUG] [app] Ending request: urn:request:0147066a-c96d-470e-a92b-c230cac8630b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0147066a-c96d-470e-a92b-c230cac8630b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:00,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:11:00,640 [285] [DEBUG] [app] Starting request: urn:request:f2d1d56b-cf64-4064-894b-9e16fed8dda2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:00,640 [285] [DEBUG] [app] Ending request: urn:request:f2d1d56b-cf64-4064-894b-9e16fed8dda2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f2d1d56b-cf64-4064-894b-9e16fed8dda2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:11:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:00,661 [286] [DEBUG] [app] Ending request: urn:request:5efa1270-1ed3-42ba-ad29-1a33d75c1524 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5efa1270-1ed3-42ba-ad29-1a33d75c1524', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:11:00,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:11:00,669 [282] [DEBUG] [app] Ending request: urn:request:044cb297-54ba-4b3f-beb5-d18a19980e19 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:044cb297-54ba-4b3f-beb5-d18a19980e19', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:00,669 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:11:00,669 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.038 118 0.038)
gcworker stdout | 2024-04-22 04:11:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:11:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:31.842666+00:00 (in 29.999460 seconds)
gcworker stdout | 2024-04-22 04:11:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:31 UTC)" (scheduled at 2024-04-22 04:11:01.842666+00:00)
gcworker stdout | 2024-04-22 04:11:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549461843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:11:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:11:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:11:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:11:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:11:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:09.033206+00:00 (in 6.997071 seconds)
repositorygcworker stdout | 2024-04-22 04:11:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:02 UTC)" (scheduled at 2024-04-22 04:11:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:11:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:11:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:11:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:12.430973+00:00 (in 9.999476 seconds)
notificationworker stdout | 2024-04-22 04:11:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:12 UTC)" (scheduled at 2024-04-22 04:11:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:11:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:11:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 2, 431930), True, datetime.datetime(2024, 4, 22, 4, 11, 2, 431930), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:11:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:11:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:11:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:11:05,544 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:11:06,812 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:11:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:11:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:02.035613+00:00 (in 53.001967 seconds)
repositorygcworker stdout | 2024-04-22 04:11:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:09 UTC)" (scheduled at 2024-04-22 04:11:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:11:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:11:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 9, 34021), True, datetime.datetime(2024, 4, 22, 4, 11, 9, 34021), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:11:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:11:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:11:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:11:09,389 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:11:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:11:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:40.238173+00:00 (in 29.999433 seconds)
autopruneworker stdout | 2024-04-22 04:11:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:40 UTC)" (scheduled at 2024-04-22 04:11:10.238173+00:00)
namespacegcworker stdout | 2024-04-22 04:11:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:11:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:30.247852+00:00 (in 19.998735 seconds)
namespacegcworker stdout | 2024-04-22 04:11:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:10 UTC)" (scheduled at 2024-04-22 04:11:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:11:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:11:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:11:10,252 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759010251, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:11:10,257 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:11:10,257 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:11:10,257 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:11:10,430 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:11:10,703 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:11:10,912 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:11:11,437 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:11:12,077 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:11:12,316 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:11:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:22.430973+00:00 (in 9.999509 seconds)
notificationworker stdout | 2024-04-22 04:11:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:22 UTC)" (scheduled at 2024-04-22 04:11:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:11:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:11:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 12, 431888), True, datetime.datetime(2024, 4, 22, 4, 11, 12, 431888), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:11:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:11:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:11:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:11:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:11:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:27.806778+00:00 (in 14.999463 seconds)
quotatotalworker stdout | 2024-04-22 04:11:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:27 UTC)" (scheduled at 2024-04-22 04:11:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:11:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:11:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:11:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:11:13,584 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:11:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:11:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:17.065745+00:00 (in 2.998169 seconds)
securityworker stdout | 2024-04-22 04:11:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:44 UTC)" (scheduled at 2024-04-22 04:11:14.067050+00:00)
securityworker stdout | 2024-04-22 04:11:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:11:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:11:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:11:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:11:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:11:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:11:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:11:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:11:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:11:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:11:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 6, 14, 78025), 1, 2])
securityworker stdout | 2024-04-22 04:11:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:11:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:11:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 6, 14, 78025), 1, 2])
securityworker stdout | 2024-04-22 04:11:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:11:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:11:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:11:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:11:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:11:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:11:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:11:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:11:14,592 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:11:15,632 [284] [DEBUG] [app] Starting request: urn:request:3a31c8f0-a584-4b53-bd9d-df50736d33c0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:15,632 [286] [DEBUG] [app] Starting request: urn:request:4c3e4392-7153-4f5f-a90b-4cfaec789d10 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:11:15,635 [280] [DEBUG] [app] Starting request: urn:request:e5e47fe3-0647-4394-b7ed-fd80c6943e0f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:15,635 [283] [DEBUG] [app] Starting request: urn:request:6ec556d3-4d33-447b-8269-e621f6391229 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:15,635 [280] [DEBUG] [app] Ending request: urn:request:e5e47fe3-0647-4394-b7ed-fd80c6943e0f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e5e47fe3-0647-4394-b7ed-fd80c6943e0f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:11:15,635 [283] [DEBUG] [app] Ending request: urn:request:6ec556d3-4d33-447b-8269-e621f6391229 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6ec556d3-4d33-447b-8269-e621f6391229', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:11:15,635 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:11:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:11:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:15,638 [284] [DEBUG] [app] Starting request: urn:request:e1b81810-f457-48c1-8d4c-61eac15db227 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:15,639 [284] [DEBUG] [app] Ending request: urn:request:e1b81810-f457-48c1-8d4c-61eac15db227 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e1b81810-f457-48c1-8d4c-61eac15db227', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:15,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:11:15,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:15,640 [282] [DEBUG] [app] Starting request: urn:request:94ebc42f-e871-489d-906f-ab575c5fd139 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:15,641 [282] [DEBUG] [app] Ending request: urn:request:94ebc42f-e871-489d-906f-ab575c5fd139 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:94ebc42f-e871-489d-906f-ab575c5fd139', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:15,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:15,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:11:15,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:15,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:15,659 [286] [DEBUG] [app] Ending request: urn:request:4c3e4392-7153-4f5f-a90b-4cfaec789d10 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c3e4392-7153-4f5f-a90b-4cfaec789d10', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:11:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:11:15,661 [284] [DEBUG] [app] Ending request: urn:request:3a31c8f0-a584-4b53-bd9d-df50736d33c0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3a31c8f0-a584-4b53-bd9d-df50736d33c0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:15,661 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.031)
gunicorn-web stdout | 2024-04-22 04:11:15,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:11:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:11:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:44.067050+00:00 (in 27.000766 seconds)
securityworker stdout | 2024-04-22 04:11:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:47 UTC)" (scheduled at 2024-04-22 04:11:17.065745+00:00)
securityworker stdout | 2024-04-22 04:11:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:11:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:11:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:11:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:11:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
expiredappspecifictokenworker stdout | 2024-04-22 04:11:17,086 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
securityworker stdout | 2024-04-22 04:11:17,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:11:17,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:11:17,258 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:11:17,769 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:11:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:11:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:53.045554+00:00 (in 34.997853 seconds)
exportactionlogsworker stdout | 2024-04-22 04:11:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:18 UTC)" (scheduled at 2024-04-22 04:11:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:11:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:11:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:11:18,390 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:11:19,340 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:11:19,931 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:11:19,934 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:11:19,937 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:11:20,431 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:11:20,867 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:11:21,461 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:11:21,519 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:11:21,526 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:11:21,529 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:11:21,531 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:11:21,535 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:11:21,550 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:11:21,554 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:11:21,557 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:11:21,589 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:11:21,597 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:11:21,602 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:11:21,608 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:11:21,611 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:11:22,102 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:11:22,376 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
notificationworker stdout | 2024-04-22 04:11:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:32.430973+00:00 (in 9.999512 seconds)
notificationworker stdout | 2024-04-22 04:11:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:32 UTC)" (scheduled at 2024-04-22 04:11:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:11:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:11:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 22, 431844), True, datetime.datetime(2024, 4, 22, 4, 11, 22, 431844), 0, 'notification/%', 50, 1, 0])
builder stdout | 2024-04-22 04:11:22,439 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:11:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:11:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:11:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:32 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:11:23,288 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:11:23,335 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:11:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:11:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:24.738365+00:00 (in 59.999465 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:11:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:24 UTC)" (scheduled at 2024-04-22 04:11:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:11:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:11:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:11:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:11:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:11:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:42.806778+00:00 (in 14.999476 seconds)
quotatotalworker stdout | 2024-04-22 04:11:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:42 UTC)" (scheduled at 2024-04-22 04:11:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:11:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:11:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:11:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:11:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:11:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:58.130973+00:00 (in 29.999485 seconds)
buildlogsarchiver stdout | 2024-04-22 04:11:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:58 UTC)" (scheduled at 2024-04-22 04:11:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:11:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 11, 28, 131845), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:11:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:11:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:11:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:11:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:31.933721+00:00 (in 2.002313 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:29 UTC)" (scheduled at 2024-04-22 04:11:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 29, 931847), True, datetime.datetime(2024, 4, 22, 4, 11, 29, 931847), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:11:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:11:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:11:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:47.245913+00:00 (in 16.997618 seconds)
namespacegcworker stdout | 2024-04-22 04:11:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:16:30 UTC)" (scheduled at 2024-04-22 04:11:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 04:11:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 11, 30, 248518), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:11:30,262 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 11, 30, 248518), True, datetime.datetime(2024, 4, 22, 4, 11, 30, 248518), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:11:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 11, 30, 248518), True, datetime.datetime(2024, 4, 22, 4, 11, 30, 248518), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 4, 11, 30, 248518), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:11:30,270 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:11:30,270 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:16:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:11:30,632 [284] [DEBUG] [app] Starting request: urn:request:ddc98aea-67ef-4150-adf4-1892405188c7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:30,632 [286] [DEBUG] [app] Starting request: urn:request:02b6baa2-fa43-4117-be3a-9fefcc831486 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:11:30,635 [281] [DEBUG] [app] Starting request: urn:request:a074fea7-50bd-4b9e-8fdc-ef66693ba94e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:30,635 [275] [DEBUG] [app] Starting request: urn:request:ff6fbe77-090f-4cc0-ba63-03e42030cc94 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:30,636 [281] [DEBUG] [app] Ending request: urn:request:a074fea7-50bd-4b9e-8fdc-ef66693ba94e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a074fea7-50bd-4b9e-8fdc-ef66693ba94e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:11:30,636 [275] [DEBUG] [app] Ending request: urn:request:ff6fbe77-090f-4cc0-ba63-03e42030cc94 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ff6fbe77-090f-4cc0-ba63-03e42030cc94', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:11:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:11:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:11:30,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:30,638 [284] [DEBUG] [app] Starting request: urn:request:63dbdd06-1485-4302-aa06-f262e64a2cb5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:30,639 [284] [DEBUG] [app] Ending request: urn:request:63dbdd06-1485-4302-aa06-f262e64a2cb5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:63dbdd06-1485-4302-aa06-f262e64a2cb5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:30,639 [282] [DEBUG] [app] Starting request: urn:request:aa9d4972-c7e5-4b3e-b6b4-6992f0736422 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:11:30,639 [282] [DEBUG] [app] Ending request: urn:request:aa9d4972-c7e5-4b3e-b6b4-6992f0736422 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aa9d4972-c7e5-4b3e-b6b4-6992f0736422', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:30,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:11:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:30,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:30,660 [284] [DEBUG] [app] Ending request: urn:request:ddc98aea-67ef-4150-adf4-1892405188c7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddc98aea-67ef-4150-adf4-1892405188c7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:30,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:11:30,660 [286] [DEBUG] [app] Ending request: urn:request:02b6baa2-fa43-4117-be3a-9fefcc831486 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:02b6baa2-fa43-4117-be3a-9fefcc831486', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:30,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:11:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:11:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:11:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:11:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:01.842666+00:00 (in 29.999476 seconds)
gcworker stdout | 2024-04-22 04:11:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:01 UTC)" (scheduled at 2024-04-22 04:11:31.842666+00:00)
gcworker stdout | 2024-04-22 04:11:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549491843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:11:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:11:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:11:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:11:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:11:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:29.931002+00:00 (in 57.996775 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:11:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:31 UTC)" (scheduled at 2024-04-22 04:11:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:11:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:11:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:11:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:42.430973+00:00 (in 9.999480 seconds)
notificationworker stdout | 2024-04-22 04:11:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:42 UTC)" (scheduled at 2024-04-22 04:11:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:11:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:11:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 32, 431900), True, datetime.datetime(2024, 4, 22, 4, 11, 32, 431900), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:11:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:11:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:11:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:11:35,574 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:11:36,848 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:11:39,426 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:11:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:11:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:10.238173+00:00 (in 29.999420 seconds)
autopruneworker stdout | 2024-04-22 04:11:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:10 UTC)" (scheduled at 2024-04-22 04:11:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:11:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759040247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:11:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:11:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:11:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:11:40,437 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:11:40,740 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:11:40,938 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:11:41,453 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:11:42,098 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:11:42,332 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:11:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:52.430973+00:00 (in 9.999475 seconds)
notificationworker stdout | 2024-04-22 04:11:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:52 UTC)" (scheduled at 2024-04-22 04:11:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:11:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:11:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 42, 431906), True, datetime.datetime(2024, 4, 22, 4, 11, 42, 431906), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:11:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:11:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:11:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:11:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:11:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:11:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:57.806778+00:00 (in 14.999459 seconds)
quotatotalworker stdout | 2024-04-22 04:11:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:57 UTC)" (scheduled at 2024-04-22 04:11:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:11:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:11:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:11:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:11:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:11:43,596 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:11:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:11:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:47.065745+00:00 (in 2.998156 seconds)
securityworker stdout | 2024-04-22 04:11:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:14 UTC)" (scheduled at 2024-04-22 04:11:44.067050+00:00)
securityworker stdout | 2024-04-22 04:11:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:11:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:11:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:11:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:11:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:11:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:11:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:11:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:11:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:11:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:11:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 6, 44, 78034), 1, 2])
securityworker stdout | 2024-04-22 04:11:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:11:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:11:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 6, 44, 78034), 1, 2])
securityworker stdout | 2024-04-22 04:11:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:11:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:11:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:11:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:11:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:11:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:11:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:11:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:11:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:11:44,629 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:11:45,632 [284] [DEBUG] [app] Starting request: urn:request:bb112965-c72e-46b7-a158-a0eabdf2ae9e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:45,632 [286] [DEBUG] [app] Starting request: urn:request:128d8349-8f35-4bf7-bdd1-ce1d9c3a9621 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:11:45,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:11:45,636 [275] [DEBUG] [app] Starting request: urn:request:6388f9e6-7627-4244-af02-2b897c1e597f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:45,636 [278] [DEBUG] [app] Starting request: urn:request:afab9591-98a3-4881-80cd-80fed6c5f6f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:11:45,636 [278] [DEBUG] [app] Ending request: urn:request:afab9591-98a3-4881-80cd-80fed6c5f6f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:afab9591-98a3-4881-80cd-80fed6c5f6f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:11:45,636 [275] [DEBUG] [app] Ending request: urn:request:6388f9e6-7627-4244-af02-2b897c1e597f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6388f9e6-7627-4244-af02-2b897c1e597f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:11:45,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:11:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:11:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:45,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:45,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:11:45,639 [282] [DEBUG] [app] Starting request: urn:request:3cc78938-9cce-4dcf-bca5-2bc67491e2d4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:45,639 [285] [DEBUG] [app] Starting request: urn:request:c590c2b1-e934-45bb-a2dc-685ff551339f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:11:45,640 [282] [DEBUG] [app] Ending request: urn:request:3cc78938-9cce-4dcf-bca5-2bc67491e2d4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3cc78938-9cce-4dcf-bca5-2bc67491e2d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:11:45,640 [285] [DEBUG] [app] Ending request: urn:request:c590c2b1-e934-45bb-a2dc-685ff551339f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c590c2b1-e934-45bb-a2dc-685ff551339f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:11:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:11:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:11:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:11:45,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:11:45,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:11:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:45,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:11:45,660 [286] [DEBUG] [app] Ending request: urn:request:128d8349-8f35-4bf7-bdd1-ce1d9c3a9621 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:128d8349-8f35-4bf7-bdd1-ce1d9c3a9621', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:45,660 [284] [DEBUG] [app] Ending request: urn:request:bb112965-c72e-46b7-a158-a0eabdf2ae9e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb112965-c72e-46b7-a158-a0eabdf2ae9e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:11:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:11:45,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:11:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:11:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:11:45,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:11:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:11:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:11:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:14.067050+00:00 (in 27.000741 seconds)
securityworker stdout | 2024-04-22 04:11:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:17 UTC)" (scheduled at 2024-04-22 04:11:47.065745+00:00)
securityworker stdout | 2024-04-22 04:11:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:11:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:11:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:11:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:11:47,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:11:47,086 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:11:47,086 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:17 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:11:47,122 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
namespacegcworker stdout | 2024-04-22 04:11:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:11:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:10.248668+00:00 (in 23.002276 seconds)
namespacegcworker stdout | 2024-04-22 04:11:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:47 UTC)" (scheduled at 2024-04-22 04:11:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:11:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:11:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 47, 246655), True, datetime.datetime(2024, 4, 22, 4, 11, 47, 246655), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:11:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:11:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:11:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:11:47,268 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:11:47,806 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:11:48,411 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:11:49,373 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:11:49,939 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:11:49,942 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:11:49,944 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:11:50,468 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:11:50,904 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:11:51,471 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:11:51,531 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:11:51,537 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:11:51,540 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:11:51,543 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:11:51,546 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:11:51,559 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:11:51,563 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:11:51,566 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:11:51,595 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:11:51,604 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:11:51,610 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:11:51,614 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:11:51,617 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:11:52,138 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:11:52,385 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
notificationworker stdout | 2024-04-22 04:11:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:11:54.433683+00:00 (in 2.002209 seconds)
notificationworker stdout | 2024-04-22 04:11:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:02 UTC)" (scheduled at 2024-04-22 04:11:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:11:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:11:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 52, 431886), True, datetime.datetime(2024, 4, 22, 4, 11, 52, 431886), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:11:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:11:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:11:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:02 UTC)" executed successfully
builder stdout | 2024-04-22 04:11:52,474 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:11:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:11:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:18.047111+00:00 (in 25.001066 seconds)
exportactionlogsworker stdout | 2024-04-22 04:11:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:53 UTC)" (scheduled at 2024-04-22 04:11:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:11:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:11:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 11, 53, 46464), True, datetime.datetime(2024, 4, 22, 4, 11, 53, 46464), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:11:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:11:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:11:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:11:53,325 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:11:53,367 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:11:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:11:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:02.430973+00:00 (in 7.996769 seconds)
notificationworker stdout | 2024-04-22 04:11:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:54 UTC)" (scheduled at 2024-04-22 04:11:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:11:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:11:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:12:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:11:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:11:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:12.806778+00:00 (in 14.999484 seconds)
quotatotalworker stdout | 2024-04-22 04:11:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:12 UTC)" (scheduled at 2024-04-22 04:11:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:11:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:11:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:11:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:11:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:11:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:28.130973+00:00 (in 29.999482 seconds)
buildlogsarchiver stdout | 2024-04-22 04:11:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:28 UTC)" (scheduled at 2024-04-22 04:11:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:11:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 11, 58, 131747), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:11:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:11:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:11:58,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:12:00,632 [282] [DEBUG] [app] Starting request: urn:request:326c4bc3-c3b5-45aa-95e1-a7ca76b9568e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:00,632 [285] [DEBUG] [app] Starting request: urn:request:7523bdc6-b4a0-440b-ac90-10b7697e4c7d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:12:00,636 [275] [DEBUG] [app] Starting request: urn:request:aca26243-86dc-47f3-b0ee-33628f1162d4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:00,636 [281] [DEBUG] [app] Starting request: urn:request:f042a224-0f5f-4287-bd9a-5abe90b2bd7f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:00,636 [281] [DEBUG] [app] Ending request: urn:request:f042a224-0f5f-4287-bd9a-5abe90b2bd7f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f042a224-0f5f-4287-bd9a-5abe90b2bd7f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:00,636 [275] [DEBUG] [app] Ending request: urn:request:aca26243-86dc-47f3-b0ee-33628f1162d4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:aca26243-86dc-47f3-b0ee-33628f1162d4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:12:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:12:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:00,639 [282] [DEBUG] [app] Starting request: urn:request:1e6d194f-7834-4ddf-a2d8-ee85ad22e2b1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:00,639 [285] [DEBUG] [app] Starting request: urn:request:914409ff-3ac7-4db2-aad4-6c3371bd9ca8 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:00,639 [282] [DEBUG] [app] Ending request: urn:request:1e6d194f-7834-4ddf-a2d8-ee85ad22e2b1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1e6d194f-7834-4ddf-a2d8-ee85ad22e2b1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:00,639 [285] [DEBUG] [app] Ending request: urn:request:914409ff-3ac7-4db2-aad4-6c3371bd9ca8 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:914409ff-3ac7-4db2-aad4-6c3371bd9ca8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:00,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:00,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:12:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:00,660 [282] [DEBUG] [app] Ending request: urn:request:326c4bc3-c3b5-45aa-95e1-a7ca76b9568e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:326c4bc3-c3b5-45aa-95e1-a7ca76b9568e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:12:00,660 [285] [DEBUG] [app] Ending request: urn:request:7523bdc6-b4a0-440b-ac90-10b7697e4c7d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7523bdc6-b4a0-440b-ac90-10b7697e4c7d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:12:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:12:00,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:12:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:12:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:31.842666+00:00 (in 29.999442 seconds)
gcworker stdout | 2024-04-22 04:12:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:31 UTC)" (scheduled at 2024-04-22 04:12:01.842666+00:00)
gcworker stdout | 2024-04-22 04:12:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549521843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:12:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:12:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:12:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:12:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:12:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:09.033206+00:00 (in 6.997048 seconds)
repositorygcworker stdout | 2024-04-22 04:12:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:02 UTC)" (scheduled at 2024-04-22 04:12:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:12:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:12:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:12:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:12.430973+00:00 (in 9.999496 seconds)
notificationworker stdout | 2024-04-22 04:12:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:12 UTC)" (scheduled at 2024-04-22 04:12:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:12:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:12:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 2, 431850), True, datetime.datetime(2024, 4, 22, 4, 12, 2, 431850), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:12:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:12 UTC)" executed successfully
gunicorn-registry stdout | 2024-04-22 04:12:03,349 [275] [DEBUG] [app] Starting request: urn:request:96f4ede4-b0a0-4af3-8c21-f820ec081501 (/v2/build_test/pushno/blobs/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066) {'X-Forwarded-For': '123.123.45.4, 10.131.0.9'}
gunicorn-registry stdout | 2024-04-22 04:12:03,349 [275] [DEBUG] [auth.registry_jwt_auth] Called with params: (), {'upload_uuid': '21d746f0-ec3a-4869-92ea-3cf4dcfc9066', 'namespace_name': 'build_test', 'repo_name': 'pushno'}
gunicorn-registry stdout | 2024-04-22 04:12:03,349 [275] [DEBUG] [auth.registry_jwt_auth] Validating auth header: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:03,349 [275] [DEBUG] [util.security.registry_jwt] encoded JWT: eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:03,351 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2024, 4, 22, 4, 12, 3, 350226), None, 'quay', 'quay', datetime.datetime(2024, 4, 22, 4, 12, 3, 350257), 'quay', None, datetime.datetime(2024, 4, 21, 4, 12, 3, 350278), datetime.datetime(2024, 4, 15, 4, 12, 3, 350288), None])
gunicorn-registry stdout | 2024-04-22 04:12:03,351 [275] [DEBUG] [peewee.pool] No connection available in pool.
gunicorn-registry stdout | 2024-04-22 04:12:03,360 [275] [DEBUG] [peewee.pool] Created new connection 140138499064512.
gunicorn-registry stdout | 2024-04-22 04:12:03,369 [275] [DEBUG] [auth.permissions] Identity loaded: <Identity id="whuquay" auth_type="signed_jwt" provides={_ResourceNeed(type='repository', namespace='build_test', name='pushno', role='write')}>
gunicorn-registry stdout | 2024-04-22 04:12:03,369 [275] [DEBUG] [auth.permissions] Loaded signed_jwt identity for: whuquay
gunicorn-registry stdout | 2024-04-22 04:12:03,369 [275] [DEBUG] [auth.registry_jwt_auth] Identity changed to whuquay
gunicorn-registry stdout | 2024-04-22 04:12:03,369 [275] [DEBUG] [auth.registry_jwt_auth] Auth context set to {'version': 2, 'entity_kind': 'user', 'entity_reference': 'a1442566-a2e4-4fbd-86d7-cdbe44891b3e', 'kind': 'user', 'user': 'whuquay', 'com.apostille.roots': {'build_test/pushno': '$disabled'}, 'com.apostille.root': '$disabled'}
gunicorn-registry stdout | 2024-04-22 04:12:03,369 [275] [DEBUG] [endpoints.v2] Checking permission <class 'auth.permissions.ModifyRepositoryPermission'> for repo: build_test/pushno
gunicorn-registry stdout | 2024-04-22 04:12:03,371 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'pushno', 3, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,376 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,380 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,384 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1"', [])
gunicorn-registry stdout | 2024-04-22 04:12:03,388 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['public', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,391 [275] [DEBUG] [peewee] ('SELECT Sum("t1"."byte_count") AS "size_bytes" FROM "blobupload" AS "t1" WHERE ("t1"."repository_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,396 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,400 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,404 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,408 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test'])
gunicorn-registry stdout | 2024-04-22 04:12:03,412 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['21d746f0-ec3a-4869-92ea-3cf4dcfc9066', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,416 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:03,416 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:03,417 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['21d746f0-ec3a-4869-92ea-3cf4dcfc9066', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,421 [275] [DEBUG] [peewee] ('UPDATE "blobupload" SET "repository_id" = %s, "uuid" = %s, "byte_count" = %s, "sha_state" = %s, "location_id" = %s, "storage_metadata" = %s, "chunk_count" = %s, "uncompressed_byte_count" = %s, "created" = %s, "piece_sha_state" = %s, "piece_hashes" = %s WHERE ("blobupload"."id" = %s)', [1, '21d746f0-ec3a-4869-92ea-3cf4dcfc9066', 2798889, 'gASVsgAAAAAAAACMBnJlaGFzaJSMBnNoYTI1NpSTlCmBlH2UKIwEbmFtZZSMBnNoYTI1NpSMB21kX2RhdGGUQ3jpqeXO1/No27w5r2wN3PI3NLb5luCuv0Ctihdu+R4ha0ipVQEAAAAAa05r/ON+3znp+T90OPxBfO3/a01rWtOaTpb+/wAAAP//DdVvXQCsWAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAACkAAAAgAAAAAAAAAAAAAACUdWIu', 9, '{"blocks": ["00000"], "content-type": "application/octet-stream"}', 2, 5811200, datetime.datetime(2024, 4, 22, 4, 12, 0, 681574), None, None, 1])
gunicorn-registry stdout | 2024-04-22 04:12:03,427 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:03,427 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:03,429 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/24/2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8'
gunicorn-registry stdout | Request method: 'HEAD'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '79549ce6-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:03,432 [275] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): whusc1.blob.core.windows.net:443
gunicorn-registry stdout | 2024-04-22 04:12:03,479 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "HEAD /whusc1container/quaydata/sha256/24/2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8 HTTP/1.1" 404 0
gunicorn-registry stdout | 2024-04-22 04:12:03,480 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 404
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Transfer-Encoding': 'chunked'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a5c3a-401e-0020-746b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '79549ce6-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-error-code': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:03,481 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066?comp=REDACTED'
gunicorn-registry stdout | Request method: 'PUT'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'Content-Type': 'application/xml; charset=utf-8'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-blob-content-type': 'REDACTED'
gunicorn-registry stdout |     'Content-Length': '87'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '795ca6f2-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | A body is sent with the request
gunicorn-registry stdout | 2024-04-22 04:12:03,491 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "PUT /whusc1container/quaydata/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066?comp=blocklist HTTP/1.1" 201 0
gunicorn-registry stdout | 2024-04-22 04:12:03,492 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 201
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout |     'ETag': '"0x8DC62825D917EA7"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a5c47-401e-0020-7f6b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '795ca6f2-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-content-crc64': 'REDACTED'
gunicorn-registry stdout |     'x-ms-request-server-encrypted': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:03,493 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/24/2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8'
gunicorn-registry stdout | Request method: 'PUT'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-copy-source': 'REDACTED'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '795e6a8c-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:03,540 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "PUT /whusc1container/quaydata/sha256/24/2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8 HTTP/1.1" 202 0
gunicorn-registry stdout | 2024-04-22 04:12:03,541 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 202
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout |     'ETag': '"0x8DC62825D98F7A0"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a5c50-401e-0020-066b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '795e6a8c-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-id': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-status': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:03,542 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/24/2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8'
gunicorn-registry stdout | Request method: 'HEAD'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7965e44c-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:03,546 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "HEAD /whusc1container/quaydata/sha256/24/2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8 HTTP/1.1" 200 0
gunicorn-registry stdout | 2024-04-22 04:12:03,546 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 200
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '2798889'
gunicorn-registry stdout |     'Content-Type': 'application/octet-stream'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout |     'Accept-Ranges': 'REDACTED'
gunicorn-registry stdout |     'ETag': '"0x8DC62825D98F7A0"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a5c74-401e-0020-266b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7965e44c-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-creation-time': 'REDACTED'
gunicorn-registry stdout |     'x-ms-lease-status': 'REDACTED'
gunicorn-registry stdout |     'x-ms-lease-state': 'REDACTED'
gunicorn-registry stdout |     'x-ms-blob-type': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-id': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-source': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-status': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-progress': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-completion-time': 'REDACTED'
gunicorn-registry stdout |     'x-ms-server-encrypted': 'REDACTED'
gunicorn-registry stdout |     'x-ms-access-tier': 'REDACTED'
gunicorn-registry stdout |     'x-ms-access-tier-inferred': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:03,547 [275] [DEBUG] [storage.azurestorage] Deleting chunked upload 21d746f0-ec3a-4869-92ea-3cf4dcfc9066 at path quaydata/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066
gunicorn-registry stdout | 2024-04-22 04:12:03,548 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066'
gunicorn-registry stdout | Request method: 'DELETE'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7966d17c-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:03,554 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "DELETE /whusc1container/quaydata/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066 HTTP/1.1" 202 0
gunicorn-registry stdout | 2024-04-22 04:12:03,555 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 202
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a5c7d-401e-0020-2f6b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7966d17c-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-delete-type-permanent': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:03 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:03,556 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['21d746f0-ec3a-4869-92ea-3cf4dcfc9066', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,559 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "imagestoragelocation" AS "t1"', [])
gunicorn-registry stdout | 2024-04-22 04:12:03,561 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."image_size", "t1"."uncompressed_size", "t1"."uploading", "t1"."cas_path", "t1"."content_checksum" FROM "imagestorage" AS "t1" WHERE ("t1"."content_checksum" = %s) LIMIT %s OFFSET %s', ['sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,563 [275] [DEBUG] [peewee] ('INSERT INTO "imagestorage" ("uuid", "image_size", "uncompressed_size", "uploading", "cas_path", "content_checksum") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "imagestorage"."id"', ['75e7b2f0-d585-436e-9e94-469a3ed9598e', 2798889, 5811200, True, True, 'sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8'])
gunicorn-registry stdout | 2024-04-22 04:12:03,565 [275] [DEBUG] [peewee] ('INSERT INTO "imagestorageplacement" ("storage_id", "location_id") VALUES (%s, %s) RETURNING "imagestorageplacement"."id"', [1, 9])
gunicorn-registry stdout | 2024-04-22 04:12:03,568 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:03,569 [275] [DEBUG] [peewee] ('INSERT INTO "uploadedblob" ("repository_id", "blob_id", "uploaded_at", "expires_at") VALUES (%s, %s, %s, %s) RETURNING "uploadedblob"."id"', [1, 1, datetime.datetime(2024, 4, 22, 4, 12, 3, 569478), datetime.datetime(2024, 4, 22, 5, 12, 3, 569464)])
gunicorn-registry stdout | 2024-04-22 04:12:03,571 [275] [DEBUG] [peewee] ('DELETE FROM "blobupload" WHERE ("blobupload"."id" = %s)', [1])
gunicorn-registry stdout | 2024-04-22 04:12:03,578 [275] [DEBUG] [app] Ending request: urn:request:96f4ede4-b0a0-4af3-8c21-f820ec081501 (/v2/build_test/pushno/blobs/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066) {'endpoint': 'v2.monolithic_upload_or_last_chunk', 'request_id': 'urn:request:96f4ede4-b0a0-4af3-8c21-f820ec081501', 'remote_addr': '10.131.0.9', 'http_method': 'PUT', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/v2/build_test/pushno/blobs/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066?digest=sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8', 'path': '/v2/build_test/pushno/blobs/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066', 'parameters': {'digest': 'sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8'}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'skopeo/1.14.2'}
gunicorn-registry stdout | 2024-04-22 04:12:03,578 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:03,578 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
nginx stdout | 10.131.0.9 (-) - - [22/Apr/2024:04:12:03 +0000] "PUT /v2/build_test/pushno/blobs/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066?digest=sha256%3A2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8 HTTP/1.1" 201 0 "-" "skopeo/1.14.2" (0.231 1847 0.231)
gunicorn-registry stdout | 2024-04-22 04:12:03,579 [275] [INFO] [gunicorn.access] 10.131.0.9 - - [22/Apr/2024:04:12:03 +0000] "PUT /v2/build_test/pushno/blobs/uploads/21d746f0-ec3a-4869-92ea-3cf4dcfc9066?digest=sha256%3A2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8 HTTP/1.1" 201 0 "-" "skopeo/1.14.2"
gunicorn-registry stdout | 2024-04-22 04:12:05,294 [280] [DEBUG] [app] Starting request: urn:request:9d404f08-31b7-4e33-8ec5-1a748d02c210 (/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad) {'X-Forwarded-For': '123.123.45.4, 10.131.0.9'}
gunicorn-registry stdout | 2024-04-22 04:12:05,294 [280] [DEBUG] [auth.registry_jwt_auth] Called with params: (), {'upload_uuid': '55f422d1-cddc-42e4-9f13-76f06e2852ad', 'namespace_name': 'build_test', 'repo_name': 'pushno'}
gunicorn-registry stdout | 2024-04-22 04:12:05,294 [280] [DEBUG] [auth.registry_jwt_auth] Validating auth header: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:05,294 [280] [DEBUG] [util.security.registry_jwt] encoded JWT: eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:05,296 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE ((((NOT ("t1"."approval_id" IS %s) AND (("t1"."expiration_date" > %s) OR ("t1"."expiration_date" IS %s))) AND ("t1"."service" = %s)) AND (NOT (("t1"."service" = %s) AND ("t1"."expiration_date" <= %s)) OR NOT ((("t1"."service" = %s) AND ("t1"."approval_id" IS %s)) AND ("t1"."created_date" <= %s)))) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s)))', [None, datetime.datetime(2024, 4, 22, 4, 12, 5, 295644), None, 'quay', 'quay', datetime.datetime(2024, 4, 22, 4, 12, 5, 295670), 'quay', None, datetime.datetime(2024, 4, 21, 4, 12, 5, 295687), datetime.datetime(2024, 4, 15, 4, 12, 5, 295698), None])
gunicorn-registry stdout | 2024-04-22 04:12:05,296 [280] [DEBUG] [peewee.pool] No connection available in pool.
gunicorn-registry stdout | 2024-04-22 04:12:05,305 [280] [DEBUG] [peewee.pool] Created new connection 140138499228352.
gunicorn-registry stdout | 2024-04-22 04:12:05,314 [280] [DEBUG] [auth.permissions] Identity loaded: <Identity id="whuquay" auth_type="signed_jwt" provides={_ResourceNeed(type='repository', namespace='build_test', name='pushno', role='write')}>
gunicorn-registry stdout | 2024-04-22 04:12:05,314 [280] [DEBUG] [auth.permissions] Loaded signed_jwt identity for: whuquay
gunicorn-registry stdout | 2024-04-22 04:12:05,314 [280] [DEBUG] [auth.registry_jwt_auth] Identity changed to whuquay
gunicorn-registry stdout | 2024-04-22 04:12:05,314 [280] [DEBUG] [auth.registry_jwt_auth] Auth context set to {'version': 2, 'entity_kind': 'user', 'entity_reference': 'a1442566-a2e4-4fbd-86d7-cdbe44891b3e', 'kind': 'user', 'user': 'whuquay', 'com.apostille.roots': {'build_test/pushno': '$disabled'}, 'com.apostille.root': '$disabled'}
gunicorn-registry stdout | 2024-04-22 04:12:05,315 [280] [DEBUG] [endpoints.v2] Checking permission <class 'auth.permissions.ModifyRepositoryPermission'> for repo: build_test/pushno
gunicorn-registry stdout | 2024-04-22 04:12:05,316 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'pushno', 3, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,321 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,324 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,328 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1"', [])
gunicorn-registry stdout | 2024-04-22 04:12:05,332 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['public', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,336 [280] [DEBUG] [peewee] ('SELECT Sum("t1"."byte_count") AS "size_bytes" FROM "blobupload" AS "t1" WHERE ("t1"."repository_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,340 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,344 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,348 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,352 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test'])
gunicorn-registry stdout | 2024-04-22 04:12:05,356 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55f422d1-cddc-42e4-9f13-76f06e2852ad', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,360 [280] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:05,360 [280] [DEBUG] [peewee.pool] Returning 140138499228352 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:05,362 [280] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?blockid=REDACTED&comp=REDACTED'
gunicorn-registry stdout | Request method: 'PUT'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'Content-Type': 'application/octet-stream'
gunicorn-registry stdout |     'Content-Length': '1472'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7a7ba06a-005e-11ef-a06b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Content-MD5': 'REDACTED'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | A body is sent with the request
gunicorn-registry stdout | 2024-04-22 04:12:05,365 [280] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): whusc1.blob.core.windows.net:443
gunicorn-registry stdout | 2024-04-22 04:12:05,403 [280] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "PUT /whusc1container/quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?blockid=MDAwMDA%3D&comp=block HTTP/1.1" 201 0
gunicorn-registry stdout | 2024-04-22 04:12:05,403 [280] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 201
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Content-MD5': 'REDACTED'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'aad6dd3d-701e-00e0-246b-94cfa6000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7a7ba06a-005e-11ef-a06b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-request-server-encrypted': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:04 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:05,404 [280] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55f422d1-cddc-42e4-9f13-76f06e2852ad', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,409 [280] [DEBUG] [peewee] ('UPDATE "blobupload" SET "repository_id" = %s, "uuid" = %s, "byte_count" = %s, "sha_state" = %s, "location_id" = %s, "storage_metadata" = %s, "chunk_count" = %s, "uncompressed_byte_count" = %s, "created" = %s, "piece_sha_state" = %s, "piece_hashes" = %s WHERE ("blobupload"."id" = %s)', [1, '55f422d1-cddc-42e4-9f13-76f06e2852ad', 1472, 'gASVsgAAAAAAAACMBnJlaGFzaJSMBnNoYTI1NpSTlCmBlH2UKIwEbmFtZZSMBnNoYTI1NpSMB21kX2RhdGGUQ3hRsO//TF1X7KcOgeQ9JweAHcpffPDQHWBICOn5PZE6mgAuAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAACUdWIu', 9, '{"blocks": ["00000"], "content-type": "application/octet-stream"}', 1, None, datetime.datetime(2024, 4, 22, 4, 12, 5, 25462), None, None, 2])
gunicorn-registry stdout | 2024-04-22 04:12:05,415 [280] [DEBUG] [app] Ending request: urn:request:9d404f08-31b7-4e33-8ec5-1a748d02c210 (/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad) {'endpoint': 'v2.upload_chunk', 'request_id': 'urn:request:9d404f08-31b7-4e33-8ec5-1a748d02c210', 'remote_addr': '10.131.0.9', 'http_method': 'PATCH', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad', 'path': '/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'skopeo/1.14.2'}
gunicorn-registry stdout | 2024-04-22 04:12:05,415 [280] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:05,415 [280] [DEBUG] [peewee.pool] Returning 140138499228352 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:05,415 [280] [INFO] [gunicorn.access] 10.131.0.9 - - [22/Apr/2024:04:12:05 +0000] "PATCH /v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad HTTP/1.1" 202 0 "-" "skopeo/1.14.2"
nginx stdout | 10.131.0.9 (-) - - [22/Apr/2024:04:12:05 +0000] "PATCH /v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad HTTP/1.1" 202 0 "-" "skopeo/1.14.2" (0.123 3243 0.122)
notificationworker stdout | 2024-04-22 04:12:05,611 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
gunicorn-registry stdout | 2024-04-22 04:12:05,675 [275] [DEBUG] [app] Starting request: urn:request:f66e8e8b-3c9a-41f2-8b34-ce24fbaa1d79 (/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad) {'X-Forwarded-For': '123.123.45.4, 10.131.0.9'}
gunicorn-registry stdout | 2024-04-22 04:12:05,675 [275] [DEBUG] [auth.registry_jwt_auth] Called with params: (), {'upload_uuid': '55f422d1-cddc-42e4-9f13-76f06e2852ad', 'namespace_name': 'build_test', 'repo_name': 'pushno'}
gunicorn-registry stdout | 2024-04-22 04:12:05,675 [275] [DEBUG] [auth.registry_jwt_auth] Validating auth header: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:05,675 [275] [DEBUG] [util.security.registry_jwt] encoded JWT: eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:05,676 [275] [DEBUG] [auth.permissions] Identity loaded: <Identity id="whuquay" auth_type="signed_jwt" provides={_ResourceNeed(type='repository', namespace='build_test', name='pushno', role='write')}>
gunicorn-registry stdout | 2024-04-22 04:12:05,676 [275] [DEBUG] [auth.permissions] Loaded signed_jwt identity for: whuquay
gunicorn-registry stdout | 2024-04-22 04:12:05,676 [275] [DEBUG] [auth.registry_jwt_auth] Identity changed to whuquay
gunicorn-registry stdout | 2024-04-22 04:12:05,676 [275] [DEBUG] [auth.registry_jwt_auth] Auth context set to {'version': 2, 'entity_kind': 'user', 'entity_reference': 'a1442566-a2e4-4fbd-86d7-cdbe44891b3e', 'kind': 'user', 'user': 'whuquay', 'com.apostille.roots': {'build_test/pushno': '$disabled'}, 'com.apostille.root': '$disabled'}
gunicorn-registry stdout | 2024-04-22 04:12:05,677 [275] [DEBUG] [endpoints.v2] Checking permission <class 'auth.permissions.ModifyRepositoryPermission'> for repo: build_test/pushno
gunicorn-registry stdout | 2024-04-22 04:12:05,677 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'pushno', 3, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,682 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,686 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,690 [275] [DEBUG] [peewee] ('SELECT Sum("t1"."byte_count") AS "size_bytes" FROM "blobupload" AS "t1" WHERE ("t1"."repository_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,694 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,698 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,702 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,706 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test'])
gunicorn-registry stdout | 2024-04-22 04:12:05,710 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55f422d1-cddc-42e4-9f13-76f06e2852ad', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,713 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:05,713 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:05,714 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55f422d1-cddc-42e4-9f13-76f06e2852ad', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,718 [275] [DEBUG] [peewee] ('UPDATE "blobupload" SET "repository_id" = %s, "uuid" = %s, "byte_count" = %s, "sha_state" = %s, "location_id" = %s, "storage_metadata" = %s, "chunk_count" = %s, "uncompressed_byte_count" = %s, "created" = %s, "piece_sha_state" = %s, "piece_hashes" = %s WHERE ("blobupload"."id" = %s)', [1, '55f422d1-cddc-42e4-9f13-76f06e2852ad', 1472, 'gASVsgAAAAAAAACMBnJlaGFzaJSMBnNoYTI1NpSTlCmBlH2UKIwEbmFtZZSMBnNoYTI1NpSMB21kX2RhdGGUQ3hRsO//TF1X7KcOgeQ9JweAHcpffPDQHWBICOn5PZE6mgAuAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAgAAAAAAAAAAAAAACUdWIu', 9, '{"blocks": ["00000"], "content-type": "application/octet-stream"}', 2, None, datetime.datetime(2024, 4, 22, 4, 12, 5, 25462), None, None, 2])
gunicorn-registry stdout | 2024-04-22 04:12:05,724 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:05,724 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:05,725 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'
gunicorn-registry stdout | Request method: 'HEAD'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab2ffb0-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:05,729 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "HEAD /whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 404 0
gunicorn-registry stdout | 2024-04-22 04:12:05,729 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 404
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Transfer-Encoding': 'chunked'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a61b5-401e-0020-656b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab2ffb0-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-error-code': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:05,730 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?comp=REDACTED'
gunicorn-registry stdout | Request method: 'PUT'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'Content-Type': 'application/xml; charset=utf-8'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-blob-content-type': 'REDACTED'
gunicorn-registry stdout |     'Content-Length': '87'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab3c436-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | A body is sent with the request
gunicorn-registry stdout | 2024-04-22 04:12:05,738 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "PUT /whusc1container/quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?comp=blocklist HTTP/1.1" 201 0
gunicorn-registry stdout | 2024-04-22 04:12:05,738 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 201
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout |     'ETag': '"0x8DC62825EE89F0E"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a61b7-401e-0020-676b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab3c436-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-content-crc64': 'REDACTED'
gunicorn-registry stdout |     'x-ms-request-server-encrypted': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:05,739 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'
gunicorn-registry stdout | Request method: 'PUT'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-copy-source': 'REDACTED'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab522c2-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:05,759 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "PUT /whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 202 0
gunicorn-registry stdout | 2024-04-22 04:12:05,759 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 202
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout |     'ETag': '"0x8DC62825EEB0FA7"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a61bd-401e-0020-6c6b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab522c2-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-id': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-status': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:05,760 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'
gunicorn-registry stdout | Request method: 'HEAD'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab85d3e-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:05,763 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "HEAD /whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 200 0
gunicorn-registry stdout | 2024-04-22 04:12:05,764 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 200
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '1472'
gunicorn-registry stdout |     'Content-Type': 'application/octet-stream'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout |     'Accept-Ranges': 'REDACTED'
gunicorn-registry stdout |     'ETag': '"0x8DC62825EEB0FA7"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a61cf-401e-0020-7d6b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab85d3e-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-creation-time': 'REDACTED'
gunicorn-registry stdout |     'x-ms-lease-status': 'REDACTED'
gunicorn-registry stdout |     'x-ms-lease-state': 'REDACTED'
gunicorn-registry stdout |     'x-ms-blob-type': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-id': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-source': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-status': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-progress': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-completion-time': 'REDACTED'
gunicorn-registry stdout |     'x-ms-server-encrypted': 'REDACTED'
gunicorn-registry stdout |     'x-ms-access-tier': 'REDACTED'
gunicorn-registry stdout |     'x-ms-access-tier-inferred': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:05,764 [275] [DEBUG] [storage.azurestorage] Deleting chunked upload 55f422d1-cddc-42e4-9f13-76f06e2852ad at path quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad
gunicorn-registry stdout | 2024-04-22 04:12:05,765 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad'
gunicorn-registry stdout | Request method: 'DELETE'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab91288-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:05,772 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "DELETE /whusc1container/quaydata/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad HTTP/1.1" 202 0
gunicorn-registry stdout | 2024-04-22 04:12:05,772 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 202
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '0'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a61d1-401e-0020-7f6b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7ab91288-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-delete-type-permanent': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:05,773 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."uuid", "t1"."byte_count", "t1"."sha_state", "t1"."location_id", "t1"."storage_metadata", "t1"."chunk_count", "t1"."uncompressed_byte_count", "t1"."created", "t1"."piece_sha_state", "t1"."piece_hashes", "t2"."id", "t2"."name" FROM "blobupload" AS "t1" INNER JOIN "imagestoragelocation" AS "t2" ON ("t1"."location_id" = "t2"."id") WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55f422d1-cddc-42e4-9f13-76f06e2852ad', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,776 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."image_size", "t1"."uncompressed_size", "t1"."uploading", "t1"."cas_path", "t1"."content_checksum" FROM "imagestorage" AS "t1" WHERE ("t1"."content_checksum" = %s) LIMIT %s OFFSET %s', ['sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,778 [275] [DEBUG] [peewee] ('INSERT INTO "imagestorage" ("uuid", "image_size", "uncompressed_size", "uploading", "cas_path", "content_checksum") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "imagestorage"."id"', ['c5ed443a-f1a5-4331-9336-28b15cbceeca', 1472, None, True, True, 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'])
gunicorn-registry stdout | 2024-04-22 04:12:05,779 [275] [DEBUG] [peewee] ('INSERT INTO "imagestorageplacement" ("storage_id", "location_id") VALUES (%s, %s) RETURNING "imagestorageplacement"."id"', [2, 9])
gunicorn-registry stdout | 2024-04-22 04:12:05,781 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:05,782 [275] [DEBUG] [peewee] ('INSERT INTO "uploadedblob" ("repository_id", "blob_id", "uploaded_at", "expires_at") VALUES (%s, %s, %s, %s) RETURNING "uploadedblob"."id"', [1, 2, datetime.datetime(2024, 4, 22, 4, 12, 5, 782696), datetime.datetime(2024, 4, 22, 5, 12, 5, 782688)])
gunicorn-registry stdout | 2024-04-22 04:12:05,784 [275] [DEBUG] [peewee] ('DELETE FROM "blobupload" WHERE ("blobupload"."id" = %s)', [2])
gunicorn-registry stdout | 2024-04-22 04:12:05,788 [275] [DEBUG] [app] Ending request: urn:request:f66e8e8b-3c9a-41f2-8b34-ce24fbaa1d79 (/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad) {'endpoint': 'v2.monolithic_upload_or_last_chunk', 'request_id': 'urn:request:f66e8e8b-3c9a-41f2-8b34-ce24fbaa1d79', 'remote_addr': '10.131.0.9', 'http_method': 'PUT', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?digest=sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294', 'path': '/v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad', 'parameters': {'digest': 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'skopeo/1.14.2'}
gunicorn-registry stdout | 2024-04-22 04:12:05,789 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:05,789 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
nginx stdout | 10.131.0.9 (-) - - [22/Apr/2024:04:12:05 +0000] "PUT /v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?digest=sha256%3Ae66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 201 0 "-" "skopeo/1.14.2" (0.115 1847 0.115)
gunicorn-registry stdout | 2024-04-22 04:12:05,789 [275] [INFO] [gunicorn.access] 10.131.0.9 - - [22/Apr/2024:04:12:05 +0000] "PUT /v2/build_test/pushno/blobs/uploads/55f422d1-cddc-42e4-9f13-76f06e2852ad?digest=sha256%3Ae66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 201 0 "-" "skopeo/1.14.2"
gunicorn-registry stdout | 2024-04-22 04:12:06,056 [275] [DEBUG] [app] Starting request: urn:request:23433dbc-2f19-41b3-964e-172ecf83b950 (/v2/build_test/pushno/manifests/push1) {'X-Forwarded-For': '123.123.45.4, 10.131.0.9'}
gunicorn-registry stdout | 2024-04-22 04:12:06,056 [275] [DEBUG] [auth.registry_jwt_auth] Called with params: (), {'manifest_ref': 'push1', 'namespace_name': 'build_test', 'repo_name': 'pushno'}
gunicorn-registry stdout | 2024-04-22 04:12:06,056 [275] [DEBUG] [auth.registry_jwt_auth] Validating auth header: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:06,056 [275] [DEBUG] [util.security.registry_jwt] encoded JWT: eyJhbGciOiJSUzI1NiIsImtpZCI6IkRxU21rRHNucVFUNGtYOHo5OFZkMUsyTl9OUXJVaU0tTHJVeklXUS1oc2siLCJ0eXAiOiJKV1QifQ.eyJpc3MiOiJxdWF5IiwiYXVkIjoicXVheXJlZ2lzdHJ5LXF1YXktcXVheS1lbnRlcnByaXNlLmFwcHMud2h1NDE1YXoxNS5xZS5henVyZS5kZXZjbHVzdGVyLm9wZW5zaGlmdC5jb20iLCJuYmYiOjE3MTM3NTkxMTgsImlhdCI6MTcxMzc1OTExOCwiZXhwIjoxNzEzNzYyNzE4LCJzdWIiOiJ3aHVxdWF5IiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVwb3NpdG9yeSIsIm5hbWUiOiJidWlsZF90ZXN0L3B1c2hubyIsImFjdGlvbnMiOlsicHVzaCIsInB1bGwiXX1dLCJjb250ZXh0Ijp7InZlcnNpb24iOjIsImVudGl0eV9raW5kIjoidXNlciIsImVudGl0eV9yZWZlcmVuY2UiOiJhMTQ0MjU2Ni1hMmU0LTRmYmQtODZkNy1jZGJlNDQ4OTFiM2UiLCJraW5kIjoidXNlciIsInVzZXIiOiJ3aHVxdWF5IiwiY29tLmFwb3N0aWxsZS5yb290cyI6eyJidWlsZF90ZXN0L3B1c2hubyI6IiRkaXNhYmxlZCJ9LCJjb20uYXBvc3RpbGxlLnJvb3QiOiIkZGlzYWJsZWQifX0.rMSnZyravLQkNm9Uxq_FbKIOgLUoSNTZH97zgYvomLCbQ7OGR0WMFgEs-0VFoXypa4Ht5N8VUwZXrVIFonTV3TjyaCFLciFcsKWOxKjNjL9N8vU901Ry6xdaMf5GIANH6JhmZQgaaRTuadm8Z7D0H9eQ5uUVtNNyXe-iN7fj-kJfdLFDEeW0jh6rp-TkdeDdsOjFbd34cGuyA2mfut5f0euZ9tidhg9C94UKrBKvMTIbm_vRN2y4W7GtJhiANCiOHppRrEhqimwSaa1hujhRLFExfdNalQGPk9gI5ng73TU6XkLvhe1e6bixbcywvWCGQRiv4hQK79L1-dnEIvo8Ng
gunicorn-registry stdout | 2024-04-22 04:12:06,058 [275] [DEBUG] [auth.permissions] Identity loaded: <Identity id="whuquay" auth_type="signed_jwt" provides={_ResourceNeed(type='repository', namespace='build_test', name='pushno', role='write')}>
gunicorn-registry stdout | 2024-04-22 04:12:06,058 [275] [DEBUG] [auth.permissions] Loaded signed_jwt identity for: whuquay
gunicorn-registry stdout | 2024-04-22 04:12:06,058 [275] [DEBUG] [auth.registry_jwt_auth] Identity changed to whuquay
gunicorn-registry stdout | 2024-04-22 04:12:06,058 [275] [DEBUG] [auth.registry_jwt_auth] Auth context set to {'version': 2, 'entity_kind': 'user', 'entity_reference': 'a1442566-a2e4-4fbd-86d7-cdbe44891b3e', 'kind': 'user', 'user': 'whuquay', 'com.apostille.roots': {'build_test/pushno': '$disabled'}, 'com.apostille.root': '$disabled'}
gunicorn-registry stdout | 2024-04-22 04:12:06,058 [275] [DEBUG] [endpoints.v2] Checking permission <class 'auth.permissions.ModifyRepositoryPermission'> for repo: build_test/pushno
gunicorn-registry stdout | 2024-04-22 04:12:06,059 [275] [DEBUG] [data.cache.cache_key] Loading repository lookup from cache_key: repository_lookup_build_test_pushno
gunicorn-registry stdout | 2024-04-22 04:12:06,059 [275] [DEBUG] [data.cache.impl] Checking cache for key repository_lookup_build_test_pushno
gunicorn-registry stdout | 2024-04-22 04:12:06,060 [275] [DEBUG] [data.cache.impl] Found no result in cache for key repository_lookup_build_test_pushno; calling loader
gunicorn-registry stdout | 2024-04-22 04:12:06,060 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state", "t2"."id", "t2"."uuid", "t2"."username", "t2"."password_hash", "t2"."email", "t2"."verified", "t2"."stripe_id", "t2"."organization", "t2"."robot", "t2"."invoice_email", "t2"."invalid_login_attempts", "t2"."last_invalid_login", "t2"."removed_tag_expiration_s", "t2"."enabled", "t2"."invoice_email_address", "t2"."given_name", "t2"."family_name", "t2"."company", "t2"."location", "t2"."maximum_queued_builds_count", "t2"."creation_date", "t2"."last_accessed" FROM "repository" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_user_id" = "t2"."id") WHERE ((("t2"."username" = %s) AND ("t1"."name" = %s)) AND ("t1"."state" != %s)) LIMIT %s OFFSET %s', ['build_test', 'pushno', 3, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,065 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "visibility" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,069 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "repositorykind" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,072 [275] [DEBUG] [data.cache.impl] Got loaded result for key repository_lookup_build_test_pushno: {'id': 1, 'visibility': {'id': 2, 'name': 'private'}, 'kind': {'id': 1, 'name': 'image'}, 'state': <RepositoryState.NORMAL: 0>, 'namespace_user': {'stripe_id': None}}
gunicorn-registry stdout | 2024-04-22 04:12:06,072 [275] [DEBUG] [data.cache.impl] Caching loaded result for key repository_lookup_build_test_pushno with expiration {'id': 1, 'visibility': {'id': 2, 'name': 'private'}, 'kind': {'id': 1, 'name': 'image'}, 'state': <RepositoryState.NORMAL: 0>, 'namespace_user': {'stripe_id': None}}: 120s
gunicorn-registry stdout | 2024-04-22 04:12:06,073 [275] [DEBUG] [data.cache.impl] Cached loaded result for key repository_lookup_build_test_pushno with expiration {'id': 1, 'visibility': {'id': 2, 'name': 'private'}, 'kind': {'id': 1, 'name': 'image'}, 'state': <RepositoryState.NORMAL: 0>, 'namespace_user': {'stripe_id': None}}: 120s
gunicorn-registry stdout | 2024-04-22 04:12:06,073 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."digest" = %s)) LIMIT %s OFFSET %s', [1, 'sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,078 [275] [DEBUG] [peewee] ('SELECT "t1"."uuid" FROM "imagestorage" AS "t1" INNER JOIN "uploadedblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE (("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) LIMIT %s OFFSET %s', [1, 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,082 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id", "t2"."id", "t2"."uuid", "t2"."image_size", "t2"."uncompressed_size", "t2"."uploading", "t2"."cas_path", "t2"."content_checksum" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['c5ed443a-f1a5-4331-9336-28b15cbceeca'])
gunicorn-registry stdout | 2024-04-22 04:12:06,086 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'
gunicorn-registry stdout | Request method: 'GET'
gunicorn-registry stdout | Request headers:
gunicorn-registry stdout |     'Accept': 'application/xml'
gunicorn-registry stdout |     'x-ms-range': 'REDACTED'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-date': 'REDACTED'
gunicorn-registry stdout |     'x-ms-client-request-id': '7aea1b12-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
gunicorn-registry stdout |     'Authorization': 'REDACTED'
gunicorn-registry stdout | No body was attached to the request
gunicorn-registry stdout | 2024-04-22 04:12:06,091 [275] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "GET /whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 206 1472
gunicorn-registry stdout | 2024-04-22 04:12:06,092 [275] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 206
gunicorn-registry stdout | Response headers:
gunicorn-registry stdout |     'Content-Length': '1472'
gunicorn-registry stdout |     'Content-Type': 'application/octet-stream'
gunicorn-registry stdout |     'Content-Range': 'REDACTED'
gunicorn-registry stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout |     'Accept-Ranges': 'REDACTED'
gunicorn-registry stdout |     'ETag': '"0x8DC62825EEB0FA7"'
gunicorn-registry stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
gunicorn-registry stdout |     'x-ms-request-id': 'a99a629d-401e-0020-3a6b-943798000000'
gunicorn-registry stdout |     'x-ms-client-request-id': '7aea1b12-005e-11ef-809b-0a580a810217'
gunicorn-registry stdout |     'x-ms-version': 'REDACTED'
gunicorn-registry stdout |     'x-ms-creation-time': 'REDACTED'
gunicorn-registry stdout |     'x-ms-lease-status': 'REDACTED'
gunicorn-registry stdout |     'x-ms-lease-state': 'REDACTED'
gunicorn-registry stdout |     'x-ms-blob-type': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-id': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-source': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-status': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-progress': 'REDACTED'
gunicorn-registry stdout |     'x-ms-copy-completion-time': 'REDACTED'
gunicorn-registry stdout |     'x-ms-server-encrypted': 'REDACTED'
gunicorn-registry stdout |     'Date': 'Mon, 22 Apr 2024 04:12:05 GMT'
gunicorn-registry stdout | 2024-04-22 04:12:06,094 [275] [DEBUG] [peewee] ('(SELECT * FROM (SELECT "t1"."id", "t1"."content_checksum", "t1"."image_size", "t1"."uuid", "t1"."cas_path", "t1"."uncompressed_size" FROM "imagestorage" AS "t1" INNER JOIN "uploadedblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE (("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) LIMIT %s) AS "q0") UNION ALL (SELECT * FROM (SELECT "t3"."id", "t3"."content_checksum", "t3"."image_size", "t3"."uuid", "t3"."cas_path", "t3"."uncompressed_size" FROM "imagestorage" AS "t3" INNER JOIN "uploadedblob" AS "t4" ON ("t4"."blob_id" = "t3"."id") WHERE (("t4"."repository_id" = %s) AND ("t3"."content_checksum" = %s)) LIMIT %s) AS "q1")', [1, 'sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8', 1, 1, 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294', 1])
gunicorn-registry stdout | 2024-04-22 04:12:06,098 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE (("t1"."digest" = %s) AND ("t1"."repository_id" = %s)) LIMIT %s OFFSET %s', ['sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253', 1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,102 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "mediatype" AS "t1"', [])
gunicorn-registry stdout | 2024-04-22 04:12:06,105 [275] [DEBUG] [peewee] ('INSERT INTO "manifest" ("repository_id", "digest", "media_type_id", "manifest_bytes", "config_media_type", "layers_compressed_size") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "manifest"."id"', [1, 'sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253', 15, '{\n   "schemaVersion": 2,\n   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",\n   "config": {\n      "mediaType": "application/vnd.docker.container.image.v1+json",\n      "size": 1472,\n      "digest": "sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294"\n   },\n   "layers": [\n      {\n         "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",\n         "size": 2798889,\n         "digest": "sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8"\n      }\n   ]\n}', 'application/vnd.docker.container.image.v1+json', 2798889])
gunicorn-registry stdout | 2024-04-22 04:12:06,108 [275] [DEBUG] [peewee] ('INSERT INTO "manifestblob" ("repository_id", "manifest_id", "blob_id") VALUES (%s, %s, %s), (%s, %s, %s) RETURNING "manifestblob"."id"', [1, 1, 1, 1, 1, 2])
gunicorn-registry stdout | 2024-04-22 04:12:06,110 [275] [DEBUG] [peewee] ('SELECT "t1"."namespace_user_id" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,112 [275] [DEBUG] [peewee] ('SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."id" = %s) AND "t1"."enabled") AND NOT "t1"."robot") LIMIT %s', [2, 1])
gunicorn-registry stdout | 2024-04-22 04:12:06,113 [275] [DEBUG] [peewee] ('SELECT 1 FROM "manifestblob" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") WHERE ((("t2"."namespace_user_id" = %s) AND ("t1"."blob_id" = %s)) AND ("t1"."manifest_id" != %s)) LIMIT %s', [2, 1, 1, 1])
gunicorn-registry stdout | 2024-04-22 04:12:06,115 [275] [DEBUG] [peewee] ('SELECT 1 FROM "manifestblob" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") WHERE ((("t2"."namespace_user_id" = %s) AND ("t1"."blob_id" = %s)) AND ("t1"."manifest_id" != %s)) LIMIT %s', [2, 2, 1, 1])
gunicorn-registry stdout | 2024-04-22 04:12:06,116 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,118 [275] [DEBUG] [peewee] ('UPDATE "quotanamespacesize" SET "size_bytes" = ("quotanamespacesize"."size_bytes" + %s) WHERE ("quotanamespacesize"."namespace_user_id" = %s)', [2800361, 2])
gunicorn-registry stdout | 2024-04-22 04:12:06,119 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotarepositorysize" AS "t1" WHERE ("t1"."repository_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,121 [275] [DEBUG] [peewee] ('SELECT 1 FROM "manifestblob" AS "t1" WHERE (("t1"."repository_id" = %s) AND ("t1"."manifest_id" != %s)) LIMIT %s', [1, 1, 1])
gunicorn-registry stdout | 2024-04-22 04:12:06,123 [275] [DEBUG] [peewee] ('INSERT INTO "quotarepositorysize" ("repository_id", "size_bytes", "backfill_start_ms", "backfill_complete") VALUES (%s, %s, %s, %s) RETURNING "quotarepositorysize"."id"', [1, 2800361, 0, True])
gunicorn-registry stdout | 2024-04-22 04:12:06,128 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,132 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."size_bytes", "t1"."backfill_start_ms", "t1"."backfill_complete" FROM "quotanamespacesize" AS "t1" WHERE ("t1"."namespace_user_id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,136 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,140 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."limit_bytes" FROM "userorganizationquota" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."namespace_id" = "t2"."id") WHERE ("t2"."username" = %s)', ['build_test'])
gunicorn-registry stdout | 2024-04-22 04:12:06,143 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."name" FROM "manifest" AS "t1" INNER JOIN "mediatype" AS "t2" ON ("t1"."media_type_id" = "t2"."id") WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,148 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id", "t2"."id", "t2"."repository_id", "t2"."digest", "t2"."media_type_id", "t2"."manifest_bytes", "t2"."config_media_type", "t2"."layers_compressed_size" FROM "tag" AS "t1" INNER JOIN "manifest" AS "t2" ON ("t1"."manifest_id" = "t2"."id") WHERE (((("t1"."repository_id" = %s) AND ("t1"."name" = %s)) AND (("t1"."lifetime_end_ms" IS %s) OR ("t1"."lifetime_end_ms" > %s))) AND ("t1"."hidden" = %s)) LIMIT %s OFFSET %s', [1, 'push1', None, 1713759126147, False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,152 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "tagkind" AS "t1"', [])
gunicorn-registry stdout | 2024-04-22 04:12:06,154 [275] [DEBUG] [peewee] ('INSERT INTO "tag" ("name", "repository_id", "manifest_id", "lifetime_start_ms", "lifetime_end_ms", "hidden", "reversion", "tag_kind_id") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "tag"."id"', ['push1', 1, 1, 1713759126147, None, False, False, 1])
gunicorn-registry stdout | 2024-04-22 04:12:06,159 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."uuid" = %s) AND ("t1"."organization" = %s)) LIMIT %s OFFSET %s', ['a1442566-a2e4-4fbd-86d7-cdbe44891b3e', False, 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,163 [275] [DEBUG] [util.audit] Checking publishing push_repo to the user events system
gunicorn-registry stdout | 2024-04-22 04:12:06,163 [275] [DEBUG] [util.audit] Publishing push_repo to the user events system
gunicorn-registry stdout | 2024-04-22 04:12:06,164 [275] [DEBUG] [util.audit] Logging the push_repo to analytics engine
gunicorn-registry stdout | 2024-04-22 04:12:06,164 [275] [DEBUG] [util.audit] Resolving IP address 10.131.0.9
gunicorn-registry stdout | 2024-04-22 04:12:06,166 [275] [DEBUG] [util.audit] Resolved IP address 10.131.0.9
gunicorn-registry stdout | 2024-04-22 04:12:06,166 [275] [DEBUG] [util.audit] Logging the push_repo to logs system
gunicorn-registry stdout | 2024-04-22 04:12:06,167 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0])
gunicorn-registry stdout | 2024-04-22 04:12:06,171 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "logentrykind" AS "t1"', [])
gunicorn-registry stdout | 2024-04-22 04:12:06,177 [275] [DEBUG] [peewee] ('INSERT INTO "logentry3" ("kind_id", "account_id", "performer_id", "repository_id", "datetime", "ip", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "logentry3"."id"', [42, 2, 1, 1, datetime.datetime(2024, 4, 22, 4, 12, 6, 166628), '10.131.0.9', '{"repo": "pushno", "namespace": "build_test", "user-agent": "skopeo/1.14.2", "tag": "push1", "username": "whuquay", "resolved_ip": {"provider": "internet", "service": null, "sync_token": "1645662201", "country_iso_code": null, "aws_region": null, "continent": null}}'])
gunicorn-registry stdout | 2024-04-22 04:12:06,182 [275] [DEBUG] [util.audit] Track and log of push_repo complete
gunicorn-registry stdout | 2024-04-22 04:12:06,184 [275] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'repo_push'])
gunicorn-registry stdout | 2024-04-22 04:12:06,189 [275] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/repo_push/', '{"notification_uuid": "be8728b1-b88c-4c96-82bc-47b149e585ad", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}', datetime.datetime(2024, 4, 22, 4, 12, 6, 189146), True, 5, 'cc8a91cd-7c67-40d4-b17f-e9dd39d2b8de'])
gunicorn-registry stdout | 2024-04-22 04:12:06,201 [275] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/repo_push/', '{"notification_uuid": "98da68a7-e4d8-463f-b0a3-50d4807bf84c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}', datetime.datetime(2024, 4, 22, 4, 12, 6, 189192), True, 5, '6e71b075-1dd2-400a-997a-48ab452a9799'])
gunicorn-registry stdout | 2024-04-22 04:12:06,208 [275] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/repo_push/', '{"notification_uuid": "638a67a2-7215-4c47-9e8f-1866b057fc27", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}', datetime.datetime(2024, 4, 22, 4, 12, 6, 189222), True, 5, '0de44fed-ffb1-4e91-aa40-b3df34302d4e'])
gunicorn-registry stdout | 2024-04-22 04:12:06,214 [275] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/repo_push/', '{"notification_uuid": "37a609f6-18e0-4667-9e99-ed8347122ff4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}', datetime.datetime(2024, 4, 22, 4, 12, 6, 189249), True, 5, 'd69a65e8-3260-449c-8574-91520705724a'])
gunicorn-registry stdout | 2024-04-22 04:12:06,220 [275] [DEBUG] [app] Ending request: urn:request:23433dbc-2f19-41b3-964e-172ecf83b950 (/v2/build_test/pushno/manifests/push1) {'endpoint': 'v2.write_manifest_by_tagname', 'request_id': 'urn:request:23433dbc-2f19-41b3-964e-172ecf83b950', 'remote_addr': '10.131.0.9', 'http_method': 'PUT', 'original_url': 'https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/v2/build_test/pushno/manifests/push1', 'path': '/v2/build_test/pushno/manifests/push1', 'parameters': {}, 'json_body': {'schemaVersion': 2, 'mediaType': 'application/vnd.docker.distribution.manifest.v2+json', 'config': {'mediaType': 'application/vnd.docker.container.image.v1+json', 'size': 1472, 'digest': 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'}, 'layers': [{'mediaType': 'application/vnd.docker.image.rootfs.diff.tar.gzip', 'size': 2798889, 'digest': 'sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8'}]}, 'confsha': 'ccc1cb7b', 'user-agent': 'skopeo/1.14.2'}
gunicorn-registry stdout | 2024-04-22 04:12:06,221 [275] [DEBUG] [data.database] Disconnecting from database.
gunicorn-registry stdout | 2024-04-22 04:12:06,221 [275] [DEBUG] [peewee.pool] Returning 140138499064512 to pool.
gunicorn-registry stdout | 2024-04-22 04:12:06,221 [275] [INFO] [gunicorn.access] 10.131.0.9 - - [22/Apr/2024:04:12:06 +0000] "PUT /v2/build_test/pushno/manifests/push1 HTTP/1.1" 201 2 "-" "skopeo/1.14.2"
nginx stdout | 10.131.0.9 (-) - - [22/Apr/2024:04:12:06 +0000] "PUT /v2/build_test/pushno/manifests/push1 HTTP/1.1" 201 2 "-" "skopeo/1.14.2" (0.167 2289 0.166)
gunicorn-registry stdout | 2024-04-22 04:12:06,226 [275] [DEBUG] [data.userevent] Published user event docker-cli: {'action': 'push_repo', 'repository': 'pushno', 'namespace': 'build_test'}
chunkcleanupworker stdout | 2024-04-22 04:12:06,884 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:12:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:12:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:02.035613+00:00 (in 53.001897 seconds)
repositorygcworker stdout | 2024-04-22 04:12:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:09 UTC)" (scheduled at 2024-04-22 04:12:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:12:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:12:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 9, 34174), True, datetime.datetime(2024, 4, 22, 4, 12, 9, 34174), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:12:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:12:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:12:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:12:09,462 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:12:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:12:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:40.238173+00:00 (in 29.999450 seconds)
autopruneworker stdout | 2024-04-22 04:12:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:40 UTC)" (scheduled at 2024-04-22 04:12:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:12:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759070247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:12:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:12:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:47.245913+00:00 (in 36.996786 seconds)
namespacegcworker stdout | 2024-04-22 04:12:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:10 UTC)" (scheduled at 2024-04-22 04:12:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:12:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:12:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:12:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:12:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:12:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:12:10,444 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:12:10,765 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:12:10,969 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:12:11,475 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:12:12,109 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:12:12,367 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:12:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:22.430973+00:00 (in 9.999511 seconds)
notificationworker stdout | 2024-04-22 04:12:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:22 UTC)" (scheduled at 2024-04-22 04:12:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:12:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:12:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 12, 431882), True, datetime.datetime(2024, 4, 22, 4, 12, 12, 431882), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,446 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 42, 431882), 1, 'f711d6f2-30ce-4173-bebe-69935a6cb85d', 4, 'd69a65e8-3260-449c-8574-91520705724a'])
notificationworker stdout | 2024-04-22 04:12:12,452 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "37a609f6-18e0-4667-9e99-ed8347122ff4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}
notificationworker stdout | 2024-04-22 04:12:12,452 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:12,453 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['37a609f6-18e0-4667-9e99-ed8347122ff4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,467 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationevent" AS "t1"', [])
notificationworker stdout | 2024-04-22 04:12:12,471 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "externalnotificationmethod" AS "t1"', [])
notificationworker stdout | 2024-04-22 04:12:12,475 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,484 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,488 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,492 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,498 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): hooks.slack.com:443
notificationworker stdout | 2024-04-22 04:12:12,675 [77] [DEBUG] [urllib3.connectionpool] https://hooks.slack.com:443 "POST /services/T01EV0PEJGH/B06UF0J084X/CaH6S1cDWsqFjtG85zMdLsqu HTTP/1.1" 200 2
notificationworker stdout | 2024-04-22 04:12:12,676 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['37a609f6-18e0-4667-9e99-ed8347122ff4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,681 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,685 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,689 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 4])
notificationworker stdout | 2024-04-22 04:12:12,695 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [4])
notificationworker stdout | 2024-04-22 04:12:12,702 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 12, 701928), True, datetime.datetime(2024, 4, 22, 4, 12, 12, 701928), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,706 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 42, 701928), 1, 'cc07976b-9cc1-4d00-94e5-b0251fd64327', 3, '0de44fed-ffb1-4e91-aa40-b3df34302d4e'])
notificationworker stdout | 2024-04-22 04:12:12,712 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "638a67a2-7215-4c47-9e8f-1866b057fc27", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}
notificationworker stdout | 2024-04-22 04:12:12,712 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:12,713 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['638a67a2-7215-4c47-9e8f-1866b057fc27', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:12,727 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
quotatotalworker stdout | 2024-04-22 04:12:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:12:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:27.806778+00:00 (in 14.999394 seconds)
quotatotalworker stdout | 2024-04-22 04:12:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:27 UTC)" (scheduled at 2024-04-22 04:12:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:12:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:12:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:12:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:27 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:12:13,112 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:13,114 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['638a67a2-7215-4c47-9e8f-1866b057fc27', 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,118 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,122 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,126 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 3])
notificationworker stdout | 2024-04-22 04:12:13,131 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [3])
notificationworker stdout | 2024-04-22 04:12:13,138 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 13, 137285), True, datetime.datetime(2024, 4, 22, 4, 12, 13, 137285), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,142 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 43, 137285), 1, '8aeddee6-3700-4574-8b48-0dd8979003dc', 2, '6e71b075-1dd2-400a-997a-48ab452a9799'])
notificationworker stdout | 2024-04-22 04:12:13,147 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "98da68a7-e4d8-463f-b0a3-50d4807bf84c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "updated_tags": ["push1"]}, "performer_data": {"entity_id": 1, "entity_name": "whuquay"}}
notificationworker stdout | 2024-04-22 04:12:13,147 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:13,148 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['98da68a7-e4d8-463f-b0a3-50d4807bf84c', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,162 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['whuquay', 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,166 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0])
notificationworker stdout | 2024-04-22 04:12:13,170 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."username" = %s) LIMIT %s OFFSET %s', ['build_test', 1, 0])
servicekey stdout | 2024-04-22 04:12:13,619 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:12:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:12:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:17.065745+00:00 (in 2.998129 seconds)
securityworker stdout | 2024-04-22 04:12:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:44 UTC)" (scheduled at 2024-04-22 04:12:14.067050+00:00)
securityworker stdout | 2024-04-22 04:12:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:12:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:12:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:12:14,070 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:12:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:12:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:12:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:12:14,093 [95] [INFO] [util.migrate.allocator] Found 1 candidates, processing block start: 1 end: 2
securityworker stdout | 2024-04-22 04:12:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "mediatype" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size" FROM "manifest" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,102 [95] [DEBUG] [peewee] ('SELECT "t1"."uuid" FROM "imagestorage" AS "t1" INNER JOIN "uploadedblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE (("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) LIMIT %s OFFSET %s', [1, 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294', 1, 0])
securityworker stdout | 2024-04-22 04:12:14,107 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id", "t2"."id", "t2"."uuid", "t2"."image_size", "t2"."uncompressed_size", "t2"."uploading", "t2"."cas_path", "t2"."content_checksum" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['c5ed443a-f1a5-4331-9336-28b15cbceeca'])
securityworker stdout | 2024-04-22 04:12:14,111 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "imagestoragelocation" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:14,115 [95] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294'
securityworker stdout | Request method: 'GET'
securityworker stdout | Request headers:
securityworker stdout |     'Accept': 'application/xml'
securityworker stdout |     'x-ms-range': 'REDACTED'
securityworker stdout |     'x-ms-version': 'REDACTED'
securityworker stdout |     'x-ms-date': 'REDACTED'
securityworker stdout |     'x-ms-client-request-id': '7fb33de0-005e-11ef-a3ab-0a580a810217'
securityworker stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
securityworker stdout |     'Authorization': 'REDACTED'
securityworker stdout | No body was attached to the request
securityworker stdout | 2024-04-22 04:12:14,116 [95] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): whusc1.blob.core.windows.net:443
securityworker stdout | 2024-04-22 04:12:14,151 [95] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "GET /whusc1container/quaydata/sha256/e6/e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294 HTTP/1.1" 206 1472
securityworker stdout | 2024-04-22 04:12:14,151 [95] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 206
securityworker stdout | Response headers:
securityworker stdout |     'Content-Length': '1472'
securityworker stdout |     'Content-Type': 'application/octet-stream'
securityworker stdout |     'Content-Range': 'REDACTED'
securityworker stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:05 GMT'
securityworker stdout |     'Accept-Ranges': 'REDACTED'
securityworker stdout |     'ETag': '"0x8DC62825EEB0FA7"'
securityworker stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
securityworker stdout |     'x-ms-request-id': '2df57adf-a01e-0038-116b-94e8ff000000'
securityworker stdout |     'x-ms-client-request-id': '7fb33de0-005e-11ef-a3ab-0a580a810217'
securityworker stdout |     'x-ms-version': 'REDACTED'
securityworker stdout |     'x-ms-creation-time': 'REDACTED'
securityworker stdout |     'x-ms-lease-status': 'REDACTED'
securityworker stdout |     'x-ms-lease-state': 'REDACTED'
securityworker stdout |     'x-ms-blob-type': 'REDACTED'
securityworker stdout |     'x-ms-copy-id': 'REDACTED'
securityworker stdout |     'x-ms-copy-source': 'REDACTED'
securityworker stdout |     'x-ms-copy-status': 'REDACTED'
securityworker stdout |     'x-ms-copy-progress': 'REDACTED'
securityworker stdout |     'x-ms-copy-completion-time': 'REDACTED'
securityworker stdout |     'x-ms-server-encrypted': 'REDACTED'
securityworker stdout |     'Date': 'Mon, 22 Apr 2024 04:12:14 GMT'
securityworker stdout | 2024-04-22 04:12:14,153 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."image_size", "t1"."uncompressed_size", "t1"."uploading", "t1"."cas_path", "t1"."content_checksum" FROM "imagestorage" AS "t1" WHERE ("t1"."content_checksum" = %s) LIMIT %s OFFSET %s', ['sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4', 1, 0])
securityworker stdout | 2024-04-22 04:12:14,157 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."image_size", "t1"."uncompressed_size", "t1"."uploading", "t1"."cas_path", "t1"."content_checksum" FROM "imagestorage" AS "t1" WHERE ("t1"."content_checksum" = %s) LIMIT %s OFFSET %s', ['sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4', 1, 0])
securityworker stdout | 2024-04-22 04:12:14,161 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "imagestoragelocation" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['default', 1, 0])
securityworker stdout | 2024-04-22 04:12:14,164 [95] [DEBUG] [peewee] ('INSERT INTO "imagestorage" ("uuid", "image_size", "uploading", "cas_path", "content_checksum") VALUES (%s, %s, %s, %s, %s) RETURNING "imagestorage"."id"', ['7d31a316-a865-4a60-b314-2ab5b59f2a5f', 32, True, True, 'sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'])
securityworker stdout | 2024-04-22 04:12:14,171 [95] [INFO] [azure.core.pipeline.policies.http_logging_policy] Request URL: 'https://whusc1.blob.core.windows.net/whusc1container/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4'
securityworker stdout | Request method: 'PUT'
securityworker stdout | Request headers:
securityworker stdout |     'Content-Type': 'application/octet-stream'
securityworker stdout |     'Content-Length': '32'
securityworker stdout |     'x-ms-version': 'REDACTED'
securityworker stdout |     'x-ms-blob-type': 'REDACTED'
securityworker stdout |     'x-ms-date': 'REDACTED'
securityworker stdout |     'x-ms-client-request-id': '7fbbbf88-005e-11ef-a3ab-0a580a810217'
securityworker stdout |     'User-Agent': 'azsdk-python-storage-blob/12.4.0 Python/3.9.18 (Linux-5.14.0-284.62.1.el9_2.x86_64-x86_64-with-glibc2.28)'
securityworker stdout |     'Authorization': 'REDACTED'
securityworker stdout | A body is sent with the request
securityworker stdout | 2024-04-22 04:12:14,178 [95] [DEBUG] [urllib3.connectionpool] https://whusc1.blob.core.windows.net:443 "PUT /whusc1container/quaydata/sha256/a3/a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 HTTP/1.1" 201 0
securityworker stdout | 2024-04-22 04:12:14,178 [95] [INFO] [azure.core.pipeline.policies.http_logging_policy] Response status: 201
securityworker stdout | Response headers:
securityworker stdout |     'Content-Length': '0'
securityworker stdout |     'Content-MD5': 'REDACTED'
securityworker stdout |     'Last-Modified': 'Mon, 22 Apr 2024 04:12:14 GMT'
securityworker stdout |     'ETag': '"0x8DC628263F0A43E"'
securityworker stdout |     'Server': 'Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0'
securityworker stdout |     'x-ms-request-id': '2df57aef-a01e-0038-1e6b-94e8ff000000'
securityworker stdout |     'x-ms-client-request-id': '7fbbbf88-005e-11ef-a3ab-0a580a810217'
securityworker stdout |     'x-ms-version': 'REDACTED'
securityworker stdout |     'x-ms-content-crc64': 'REDACTED'
securityworker stdout |     'x-ms-request-server-encrypted': 'REDACTED'
securityworker stdout |     'Date': 'Mon, 22 Apr 2024 04:12:14 GMT'
securityworker stdout | 2024-04-22 04:12:14,179 [95] [DEBUG] [peewee] ('INSERT INTO "imagestorageplacement" ("storage_id", "location_id") VALUES (%s, %s) RETURNING "imagestorageplacement"."id"', [3, 9])
securityworker stdout | 2024-04-22 04:12:14,186 [95] [DEBUG] [peewee] ('(SELECT * FROM (SELECT "t1"."id", "t1"."content_checksum", "t1"."image_size", "t1"."uuid", "t1"."cas_path", "t1"."uncompressed_size" FROM "imagestorage" AS "t1" INNER JOIN "manifestblob" AS "t2" ON ("t2"."blob_id" = "t1"."id") WHERE (("t2"."repository_id" = %s) AND ("t1"."content_checksum" = %s)) LIMIT %s) AS "q0") UNION ALL (SELECT * FROM (SELECT "t3"."id", "t3"."content_checksum", "t3"."image_size", "t3"."uuid", "t3"."cas_path", "t3"."uncompressed_size" FROM "imagestorage" AS "t3" INNER JOIN "manifestblob" AS "t4" ON ("t4"."blob_id" = "t3"."id") WHERE (("t4"."repository_id" = %s) AND ("t3"."content_checksum" = %s)) LIMIT %s) AS "q1")', [1, 'sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294', 1, 1, 'sha256:2408cc74d12b6cd092bb8b516ba7d5e290f485d3eb9672efc00f0583730179e8', 1])
securityworker stdout | 2024-04-22 04:12:14,191 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['75e7b2f0-d585-436e-9e94-469a3ed9598e'])
securityworker stdout | 2024-04-22 04:12:14,195 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."storage_id", "t1"."location_id" FROM "imagestorageplacement" AS "t1" INNER JOIN "imagestorage" AS "t2" ON ("t1"."storage_id" = "t2"."id") WHERE ("t2"."uuid" = %s)', ['7d31a316-a865-4a60-b314-2ab5b59f2a5f'])
securityworker stdout | 2024-04-22 04:12:14,199 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."manifest_id", "t1"."repository_id", "t1"."index_status", "t1"."error_json", "t1"."last_indexed", "t1"."indexer_hash", "t1"."indexer_version", "t1"."metadata_json" FROM "manifestsecuritystatus" AS "t1" WHERE ("t1"."manifest_id" = %s)', [1])
securityworker stdout | 2024-04-22 04:12:14,202 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,206 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,210 [95] [DEBUG] [data.secscan_model.secscan_v4_model] Indexing manifest [1] 2/pushno@sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253
securityworker stdout | 2024-04-22 04:12:14,211 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:12:14,211 [95] [DEBUG] [util.secscan.v4.api] POSTing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_report
repositorygcworker stdout | 2024-04-22 04:12:14,665 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
securityworker stdout | 2024-04-22 04:12:14,760 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "POST /indexer/api/v1/index_report HTTP/1.1" 201 None
securityworker stdout | 2024-04-22 04:12:14,763 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."manifest_id", "t1"."repository_id", "t1"."index_status", "t1"."error_json", "t1"."last_indexed", "t1"."indexer_hash", "t1"."indexer_version", "t1"."metadata_json" FROM "manifestsecuritystatus" AS "t1" WHERE ("t1"."manifest_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,768 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."repository_id", "t1"."manifest_id", "t1"."lifetime_start_ms", "t1"."lifetime_end_ms", "t1"."hidden", "t1"."reversion", "t1"."tag_kind_id", "t1"."linked_tag_id" FROM "tag" AS "t1" WHERE ("t1"."manifest_id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,772 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:12:14,772 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/matcher/api/v1/vulnerability_report/sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253
securityworker stdout | 2024-04-22 04:12:14,974 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /matcher/api/v1/vulnerability_report/sha256:4ff3ca91275773af45cb4b0834e12b7eb47d1c18f770a0b151381cd227f4c253 HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:12:14,979 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,984 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
securityworker stdout | 2024-04-22 04:12:14,989 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:14,994 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 14, 994312), True, 5, 'a9901169-74f7-4767-9436-4b49f7d5ae67'])
securityworker stdout | 2024-04-22 04:12:15,000 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 14, 994336), True, 5, 'ec7a1972-4c97-426c-b4ba-88ad25827301'])
securityworker stdout | 2024-04-22 04:12:15,005 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 14, 994351), True, 5, 'd9a91b8e-d337-485a-8493-346a5c909772'])
securityworker stdout | 2024-04-22 04:12:15,011 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 14, 994364), True, 5, '274cf96a-a2a8-4fe6-a3a8-672d359c4bbc'])
securityworker stdout | 2024-04-22 04:12:15,017 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,022 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "64181", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-30065", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 21833), True, 5, '9e7ec55c-861a-4443-b2d2-004728b5fddf'])
securityworker stdout | 2024-04-22 04:12:15,027 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "64181", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-30065", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 21860), True, 5, '4d88e942-2d61-4152-91b9-6c74559bd777'])
securityworker stdout | 2024-04-22 04:12:15,032 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "64181", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-30065", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 21876), True, 5, 'c34d745e-869d-4d0f-b82c-ccc26be5f611'])
securityworker stdout | 2024-04-22 04:12:15,038 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "64181", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-30065", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 21891), True, 5, '8b83456c-b9d3-4f60-a0ca-dff87af5a907'])
securityworker stdout | 2024-04-22 04:12:15,044 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,049 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "71141", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-37434", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 48925), True, 5, '0a8eebfd-8f64-459c-a2c9-751aa0e7c066'])
securityworker stdout | 2024-04-22 04:12:15,054 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "71141", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-37434", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 48945), True, 5, '6aadf64e-6baf-4709-bb35-9ba3d454ae11'])
securityworker stdout | 2024-04-22 04:12:15,060 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "71141", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-37434", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 48961), True, 5, 'dd8aeaf3-5972-4011-8179-1abc7727aac6'])
securityworker stdout | 2024-04-22 04:12:15,065 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "71141", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-37434", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 48974), True, 5, 'cfbf6802-47bf-43d7-aa67-82e80fd899c3'])
securityworker stdout | 2024-04-22 04:12:15,071 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,076 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68448", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0464", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 76196), True, 5, 'ae6c54d3-db1b-4746-a8ab-be2cc60f0fb0'])
securityworker stdout | 2024-04-22 04:12:15,081 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68448", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0464", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 76217), True, 5, '12eb8cb2-bebc-4752-be63-59658601be76'])
securityworker stdout | 2024-04-22 04:12:15,087 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68448", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0464", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 76232), True, 5, '3d832c4b-4e46-4df0-9b6e-c33d0a5355b9'])
securityworker stdout | 2024-04-22 04:12:15,092 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68448", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0464", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 76247), True, 5, '4f57c86c-ae3e-468c-9f4b-79967b6abc16'])
securityworker stdout | 2024-04-22 04:12:15,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,103 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68526", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0465", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 103213), True, 5, '332bdd8a-5f1f-4966-8704-ee571eb7babd'])
securityworker stdout | 2024-04-22 04:12:15,109 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68526", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0465", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 103234), True, 5, '2742810f-acd3-4ea8-92a7-438cb491dd7a'])
securityworker stdout | 2024-04-22 04:12:15,114 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68526", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0465", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 103250), True, 5, '05595694-8de4-4f7b-b742-bf137f47411d'])
securityworker stdout | 2024-04-22 04:12:15,119 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68526", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0465", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 103263), True, 5, '5d6e6c78-6000-412f-9ef2-f5701a4b33f4'])
securityworker stdout | 2024-04-22 04:12:15,126 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,130 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68523", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-2097", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 130657), True, 5, '52ad73d7-3b2d-4187-a3f4-4c58e2785e8e'])
securityworker stdout | 2024-04-22 04:12:15,136 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68523", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-2097", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 130678), True, 5, 'dda7a0f0-0fda-41e2-bb04-497ba68b8d29'])
securityworker stdout | 2024-04-22 04:12:15,149 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68523", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-2097", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 130692), True, 5, '1a1affb0-5f5d-4bde-9640-a35fa2240b6e'])
securityworker stdout | 2024-04-22 04:12:15,155 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68523", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-2097", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 130705), True, 5, 'b8974dac-89a9-42c6-b673-783954b5eb5b'])
securityworker stdout | 2024-04-22 04:12:15,161 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,166 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 166182), True, 5, '56693d59-d5db-40ad-9ad6-4c5e92d59df8'])
securityworker stdout | 2024-04-22 04:12:15,171 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 166203), True, 5, 'bd5a6145-491a-46ca-9fd2-9d0ebd113c84'])
securityworker stdout | 2024-04-22 04:12:15,177 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 166218), True, 5, '72d80dc4-639d-4e46-8a22-0b174df6c889'])
securityworker stdout | 2024-04-22 04:12:15,182 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 166232), True, 5, 'e302955d-2b1d-4e93-a89d-667ebfdc6e47'])
securityworker stdout | 2024-04-22 04:12:15,189 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,194 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68473", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0215", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 193864), True, 5, '46fd7093-ba25-4249-b82e-61bbde6222d6'])
securityworker stdout | 2024-04-22 04:12:15,200 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68473", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0215", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 193884), True, 5, 'b6af4c29-b403-4efa-9523-86a5e8fd0ddc'])
securityworker stdout | 2024-04-22 04:12:15,208 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68473", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0215", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 193899), True, 5, '31e57500-4b0d-40e6-aade-435a390631b8'])
securityworker stdout | 2024-04-22 04:12:15,215 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68473", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0215", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 193912), True, 5, '79b1e1df-3e59-4d5e-94ef-4e113137f0b0'])
securityworker stdout | 2024-04-22 04:12:15,223 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,228 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68470", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4450", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 227812), True, 5, '697fc1b2-c9a2-4221-9533-a869a4693444'])
securityworker stdout | 2024-04-22 04:12:15,236 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68470", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4450", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 227833), True, 5, 'b8974089-4200-4525-a53f-3d6b8fdc06f7'])
securityworker stdout | 2024-04-22 04:12:15,242 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68470", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4450", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 227848), True, 5, 'a02fbe20-be27-41df-9da1-d8565d7b210a'])
securityworker stdout | 2024-04-22 04:12:15,248 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68470", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4450", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 227862), True, 5, '55d5027d-a013-40a4-9b4b-5d218212e87b'])
securityworker stdout | 2024-04-22 04:12:15,261 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,266 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68467", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4304", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 266046), True, 5, '1b481ec4-92b1-4dc5-9f2e-e37e83e2230b'])
securityworker stdout | 2024-04-22 04:12:15,271 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68467", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4304", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 266067), True, 5, '3f9c911f-215e-4f16-94f0-4b81b12d0931'])
securityworker stdout | 2024-04-22 04:12:15,279 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68467", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4304", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 266104), True, 5, '64ec7c07-a490-4550-971c-09cdc62c6dc9'])
securityworker stdout | 2024-04-22 04:12:15,284 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68467", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4304", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 266120), True, 5, 'bb4da2d1-73a0-463d-a736-56b8a4d218a2'])
securityworker stdout | 2024-04-22 04:12:15,291 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,295 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 295420), True, 5, '9c9a132c-e1e9-4f54-be7b-f185cf5afb2c'])
securityworker stdout | 2024-04-22 04:12:15,301 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 295441), True, 5, '68e9d205-cb30-4a88-8623-6af1ade7ab6d'])
securityworker stdout | 2024-04-22 04:12:15,306 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 295456), True, 5, '3f93229b-d518-485b-b52a-504f377b2333'])
securityworker stdout | 2024-04-22 04:12:15,311 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 295469), True, 5, '6743155f-9143-466d-8f4e-15371887d284'])
securityworker stdout | 2024-04-22 04:12:15,317 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,322 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68495", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-2650", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 322185), True, 5, 'fc0b324b-e199-4d14-804c-2888dd1be7a5'])
securityworker stdout | 2024-04-22 04:12:15,327 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68495", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-2650", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 322207), True, 5, '79e026ee-4431-4f26-98d4-9e0efe244797'])
securityworker stdout | 2024-04-22 04:12:15,332 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68495", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-2650", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 322222), True, 5, '59f4fa58-346a-49d7-9fc9-a4a020da2536'])
securityworker stdout | 2024-04-22 04:12:15,338 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68495", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-2650", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 322235), True, 5, '98336699-3b29-4b8c-8e3e-8cfeaef88d9e'])
securityworker stdout | 2024-04-22 04:12:15,344 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") INNER JOIN "externalnotificationevent" AS "t4" ON ("t1"."event_id" = "t4"."id") WHERE ((("t3"."username" = %s) AND ("t2"."name" = %s)) AND ("t4"."name" = %s))', ['build_test', 'pushno', 'vulnerability_found'])
securityworker stdout | 2024-04-22 04:12:15,348 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68476", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0286", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 348537), True, 5, 'dce5c4f0-c75c-4b11-af6b-9c5f83b69dbd'])
securityworker stdout | 2024-04-22 04:12:15,354 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68476", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0286", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 348558), True, 5, 'af5cd2fa-8e3a-4af1-8359-c7da2c646413'])
securityworker stdout | 2024-04-22 04:12:15,359 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68476", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0286", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 348583), True, 5, 'e7df4b11-2893-466b-bee3-b832c4a582ca'])
securityworker stdout | 2024-04-22 04:12:15,365 [95] [DEBUG] [peewee] ('INSERT INTO "queueitem" ("queue_name", "body", "available_after", "available", "retries_remaining", "state_id") VALUES (%s, %s, %s, %s, %s, %s) RETURNING "queueitem"."id"', ['notification/build_test/pushno/vulnerability_found/', '{"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68476", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0286", "priority": "", "has_fix": true}}, "performer_data": {}}', datetime.datetime(2024, 4, 22, 4, 12, 15, 348596), True, 5, '9a0650a8-9c2d-4bfd-89d5-c116450c45a5'])
securityworker stdout | 2024-04-22 04:12:15,372 [95] [DEBUG] [peewee] ('DELETE FROM "manifestsecuritystatus" WHERE ("manifestsecuritystatus"."manifest_id" = %s)', [1])
securityworker stdout | 2024-04-22 04:12:15,374 [95] [DEBUG] [peewee] ('INSERT INTO "manifestsecuritystatus" ("manifest_id", "repository_id", "index_status", "error_json", "last_indexed", "indexer_hash", "indexer_version", "metadata_json") VALUES (%s, %s, %s, %s, %s, %s, %s, %s) RETURNING "manifestsecuritystatus"."id"', [1, 1, 2, '""', datetime.datetime(2024, 4, 22, 4, 12, 15, 374599), '746c21e6e713e5208118f7d54c52e391', 4, '{}'])
securityworker stdout | 2024-04-22 04:12:15,379 [95] [INFO] [util.migrate.allocator] Marking id range as completed: 1-2
securityworker stdout | 2024-04-22 04:12:15,379 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:15,380 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 7, 14, 76530), 1, 2])
securityworker stdout | 2024-04-22 04:12:15,384 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:15,384 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:15,384 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 7, 14, 76530), 1, 2])
securityworker stdout | 2024-04-22 04:12:15,388 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:15,388 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:15,388 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:15,388 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:12:15,388 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:12:15,388 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:12:15,389 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:12:15,390 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:15,390 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:12:15,390 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:44 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:12:15,632 [282] [DEBUG] [app] Starting request: urn:request:e1419aa9-8b8b-474e-a3b1-61e318d5c205 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:15,632 [285] [DEBUG] [app] Starting request: urn:request:b1329ba2-1e57-4e96-9c63-9a659421c42b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:12:15,636 [283] [DEBUG] [app] Starting request: urn:request:0bbb937f-06df-407f-b9e9-9b6e2df923d7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:15,636 [275] [DEBUG] [app] Starting request: urn:request:a02589c4-e848-4c95-89e2-b8495b99e86a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:15,636 [283] [DEBUG] [app] Ending request: urn:request:0bbb937f-06df-407f-b9e9-9b6e2df923d7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0bbb937f-06df-407f-b9e9-9b6e2df923d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:15,636 [275] [DEBUG] [app] Ending request: urn:request:a02589c4-e848-4c95-89e2-b8495b99e86a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a02589c4-e848-4c95-89e2-b8495b99e86a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:15,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:12:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:15,639 [286] [DEBUG] [app] Starting request: urn:request:9995f939-39d6-4c8d-b200-b806f41abd8d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:15,640 [286] [DEBUG] [app] Ending request: urn:request:9995f939-39d6-4c8d-b200-b806f41abd8d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9995f939-39d6-4c8d-b200-b806f41abd8d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:15,640 [284] [DEBUG] [app] Starting request: urn:request:9e2db342-0d98-49f2-98c5-8a753aeb14bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:15,640 [284] [DEBUG] [app] Ending request: urn:request:9e2db342-0d98-49f2-98c5-8a753aeb14bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9e2db342-0d98-49f2-98c5-8a753aeb14bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:12:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:15,660 [282] [DEBUG] [app] Ending request: urn:request:e1419aa9-8b8b-474e-a3b1-61e318d5c205 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e1419aa9-8b8b-474e-a3b1-61e318d5c205', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:12:15,660 [285] [DEBUG] [app] Ending request: urn:request:b1329ba2-1e57-4e96-9c63-9a659421c42b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b1329ba2-1e57-4e96-9c63-9a659421c42b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:12:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:12:15,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:12:17,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:12:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:44.067050+00:00 (in 27.000724 seconds)
securityworker stdout | 2024-04-22 04:12:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:47 UTC)" (scheduled at 2024-04-22 04:12:17.065745+00:00)
securityworker stdout | 2024-04-22 04:12:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:12:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:12:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:12:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:12:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 7, 17, 70712), 1, 2])
securityworker stdout | 2024-04-22 04:12:17,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:17,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 7, 17, 70712), 1, 2])
securityworker stdout | 2024-04-22 04:12:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:17,099 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:12:17,099 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:12:17,158 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:12:17,283 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
notificationworker stdout | 2024-04-22 04:12:17,290 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['98da68a7-e4d8-463f-b0a3-50d4807bf84c', 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,294 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,298 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,302 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 2])
notificationworker stdout | 2024-04-22 04:12:17,308 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [2])
notificationworker stdout | 2024-04-22 04:12:17,314 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 17, 313377), True, datetime.datetime(2024, 4, 22, 4, 12, 17, 313377), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,318 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 47, 313377), 1, '5cee8e81-7f6c-4ada-93ff-765a5b436fac', 11, 'c34d745e-869d-4d0f-b82c-ccc26be5f611'])
notificationworker stdout | 2024-04-22 04:12:17,324 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "64181", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-30065", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:17,324 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:17,325 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,339 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
notificationworker stdout | 2024-04-22 04:12:17,747 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:17,748 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,753 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,757 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,760 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:17,767 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [11])
notificationworker stdout | 2024-04-22 04:12:17,773 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 17, 772611), True, datetime.datetime(2024, 4, 22, 4, 12, 17, 772611), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,777 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 47, 772611), 1, '76c884e4-e13a-4e82-a559-d4b629eb7b7f', 5, 'a9901169-74f7-4767-9436-4b49f7d5ae67'])
notificationworker stdout | 2024-04-22 04:12:17,782 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:17,782 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:17,783 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,797 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE ((("t1"."name" = %s) AND ("t2"."organization" = %s)) AND ("t2"."username" = %s)) LIMIT %s', ['owners', True, 'build_test', 1])
notificationworker stdout | 2024-04-22 04:12:17,802 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "teammember" AS "t2" ON ("t2"."user_id" = "t1"."id") INNER JOIN "team" AS "t3" ON ("t2"."team_id" = "t3"."id") WHERE ("t3"."id" = %s)', [1])
notificationworker stdout | 2024-04-22 04:12:17,806 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "notificationkind" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['vulnerability_found', 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,810 [77] [DEBUG] [peewee] ('INSERT INTO "notification" ("uuid", "kind_id", "target_id", "metadata_json", "created", "dismissed", "lookup_path") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "notification"."id"', ['6462c2be-740d-4075-ab4f-27d381c698fc', 12, 1, '{"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}', datetime.datetime(2024, 4, 22, 4, 12, 17, 809839), False, None])
notificationworker stdout | 2024-04-22 04:12:17,816 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,819 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,823 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,827 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 5])
notificationworker stdout | 2024-04-22 04:12:17,832 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [5])
globalpromstats stdout | 2024-04-22 04:12:17,832 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
notificationworker stdout | 2024-04-22 04:12:17,839 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 17, 838430), True, datetime.datetime(2024, 4, 22, 4, 12, 17, 838430), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,843 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 47, 838430), 1, '4ceb538a-d66f-4364-923a-223487975213', 25, '52ad73d7-3b2d-4187-a3f4-4c58e2785e8e'])
notificationworker stdout | 2024-04-22 04:12:17,849 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68523", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-2097", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:17,849 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:17,850 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,863 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE ((("t1"."name" = %s) AND ("t2"."organization" = %s)) AND ("t2"."username" = %s)) LIMIT %s', ['owners', True, 'build_test', 1])
notificationworker stdout | 2024-04-22 04:12:17,868 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "teammember" AS "t2" ON ("t2"."user_id" = "t1"."id") INNER JOIN "team" AS "t3" ON ("t2"."team_id" = "t3"."id") WHERE ("t3"."id" = %s)', [1])
notificationworker stdout | 2024-04-22 04:12:17,872 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "notificationkind" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['vulnerability_found', 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,875 [77] [DEBUG] [peewee] ('INSERT INTO "notification" ("uuid", "kind_id", "target_id", "metadata_json", "created", "dismissed", "lookup_path") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "notification"."id"', ['565cbc7d-2776-4500-a1f6-082a90a0aeb1', 12, 1, '{"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68523", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-2097", "priority": "", "has_fix": true}}', datetime.datetime(2024, 4, 22, 4, 12, 17, 875716), False, None])
notificationworker stdout | 2024-04-22 04:12:17,881 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,885 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,889 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,892 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 5])
notificationworker stdout | 2024-04-22 04:12:17,897 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [25])
notificationworker stdout | 2024-04-22 04:12:17,903 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 17, 902953), True, datetime.datetime(2024, 4, 22, 4, 12, 17, 902953), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,908 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 47, 902953), 1, '23744d88-c22d-4eeb-b74b-645ba588751f', 55, 'e7df4b11-2893-466b-bee3-b832c4a582ca'])
notificationworker stdout | 2024-04-22 04:12:17,913 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68476", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0286", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:17,913 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:17,914 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:17,929 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
exportactionlogsworker stdout | 2024-04-22 04:12:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:12:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:23.046900+00:00 (in 4.999297 seconds)
exportactionlogsworker stdout | 2024-04-22 04:12:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:18 UTC)" (scheduled at 2024-04-22 04:12:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:12:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:12:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:18 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:12:18,348 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:18,350 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,354 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,358 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,362 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:18,367 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [55])
notificationworker stdout | 2024-04-22 04:12:18,373 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 18, 373002), True, datetime.datetime(2024, 4, 22, 4, 12, 18, 373002), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,378 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 48, 373002), 1, '75415ac1-0032-4230-b8cf-a93f546e7639', 43, '64ec7c07-a490-4550-971c-09cdc62c6dc9'])
notificationworker stdout | 2024-04-22 04:12:18,383 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68467", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4304", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:18,383 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:18,384 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,398 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
exportactionlogsworker stdout | 2024-04-22 04:12:18,444 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
notificationworker stdout | 2024-04-22 04:12:18,786 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:18,788 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,793 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,797 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,801 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:18,808 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [43])
notificationworker stdout | 2024-04-22 04:12:18,814 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 18, 813766), True, datetime.datetime(2024, 4, 22, 4, 12, 18, 813766), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,819 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 48, 813766), 1, '4709ca9d-3e1e-4ad0-a712-9ba701ac09e2', 29, '56693d59-d5db-40ad-9ad6-4c5e92d59df8'])
notificationworker stdout | 2024-04-22 04:12:18,824 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:18,824 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:18,825 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,838 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE ((("t1"."name" = %s) AND ("t2"."organization" = %s)) AND ("t2"."username" = %s)) LIMIT %s', ['owners', True, 'build_test', 1])
notificationworker stdout | 2024-04-22 04:12:18,843 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "teammember" AS "t2" ON ("t2"."user_id" = "t1"."id") INNER JOIN "team" AS "t3" ON ("t2"."team_id" = "t3"."id") WHERE ("t3"."id" = %s)', [1])
notificationworker stdout | 2024-04-22 04:12:18,847 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "notificationkind" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['vulnerability_found', 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,850 [77] [DEBUG] [peewee] ('INSERT INTO "notification" ("uuid", "kind_id", "target_id", "metadata_json", "created", "dismissed", "lookup_path") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "notification"."id"', ['70c2c07e-222c-4974-a7a0-c07333578450', 12, 1, '{"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}', datetime.datetime(2024, 4, 22, 4, 12, 18, 850673), False, None])
notificationworker stdout | 2024-04-22 04:12:18,857 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,860 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,864 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,868 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 5])
notificationworker stdout | 2024-04-22 04:12:18,873 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [29])
notificationworker stdout | 2024-04-22 04:12:18,880 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 18, 880157), True, datetime.datetime(2024, 4, 22, 4, 12, 18, 880157), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,884 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 48, 880157), 1, '5d020b1c-e285-4ef0-8dc9-cad12c1c22ae', 45, '9c9a132c-e1e9-4f54-be7b-f185cf5afb2c'])
notificationworker stdout | 2024-04-22 04:12:18,891 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:18,891 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:18,892 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,905 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."organization_id", "t1"."role_id", "t1"."description" FROM "team" AS "t1" INNER JOIN "user" AS "t2" ON ("t1"."organization_id" = "t2"."id") WHERE ((("t1"."name" = %s) AND ("t2"."organization" = %s)) AND ("t2"."username" = %s)) LIMIT %s', ['owners', True, 'build_test', 1])
notificationworker stdout | 2024-04-22 04:12:18,910 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" INNER JOIN "teammember" AS "t2" ON ("t2"."user_id" = "t1"."id") INNER JOIN "team" AS "t3" ON ("t2"."team_id" = "t3"."id") WHERE ("t3"."id" = %s)', [1])
notificationworker stdout | 2024-04-22 04:12:18,914 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "notificationkind" AS "t1" WHERE ("t1"."name" = %s) LIMIT %s OFFSET %s', ['vulnerability_found', 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,917 [77] [DEBUG] [peewee] ('INSERT INTO "notification" ("uuid", "kind_id", "target_id", "metadata_json", "created", "dismissed", "lookup_path") VALUES (%s, %s, %s, %s, %s, %s, %s) RETURNING "notification"."id"', ['2a9c0722-ff8c-422d-b197-77040f7b4972', 12, 1, '{"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}', datetime.datetime(2024, 4, 22, 4, 12, 18, 917602), False, None])
notificationworker stdout | 2024-04-22 04:12:18,923 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['f1b7c257-8b4f-4497-9b0d-ec5a64ad3ebc', 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,927 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,931 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,934 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 5])
notificationworker stdout | 2024-04-22 04:12:18,940 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [45])
notificationworker stdout | 2024-04-22 04:12:18,945 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 18, 945261), True, datetime.datetime(2024, 4, 22, 4, 12, 18, 945261), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,950 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 48, 945261), 1, '6e173ba1-6434-4502-a331-7da1d0836faa', 32, 'e302955d-2b1d-4e93-a89d-667ebfdc6e47'])
notificationworker stdout | 2024-04-22 04:12:18,955 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:18,955 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:18,956 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['27d600e3-69ac-4c25-9b0d-4999e865fe2c', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,970 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
notificationworker stdout | 2024-04-22 04:12:18,973 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = ("repositorynotification"."number_of_failures" + %s) WHERE ("repositorynotification"."uuid" = %s)', [1, '27d600e3-69ac-4c25-9b0d-4999e865fe2c'])
notificationworker stdout | 2024-04-22 04:12:18,979 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:18,979 [77] [ERROR] [workers.worker] Operation raised exception
notificationworker stdout | Traceback (most recent call last):
notificationworker stdout |   File "/quay-registry/workers/worker.py", line 86, in _operation_func
notificationworker stdout |     return operation_func()
notificationworker stdout |   File "/quay-registry/workers/queueworker.py", line 134, in poll_queue
notificationworker stdout |     self.process_queue_item(job_details)
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 45, in process_queue_item
notificationworker stdout |     raise exc
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 41, in process_queue_item
notificationworker stdout |     method_handler.perform(notification, event_handler, job_details)
notificationworker stdout |   File "/quay-registry/notifications/notificationmethod.py", line 491, in perform
notificationworker stdout |     summary = event_handler.get_summary(notification_data["event_data"], notification_data)
notificationworker stdout |   File "/quay-registry/notifications/notificationevent.py", line 250, in get_summary
notificationworker stdout |     len(event_data["tags"]),
notificationworker stdout | KeyError: 'tags'
notificationworker stdout | 2024-04-22 04:12:18,979 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:22 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:12:19,394 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:12:19,948 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:12:19,950 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:12:19,952 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:12:20,494 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:12:20,925 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:12:21,481 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:12:21,542 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:12:21,545 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:12:21,549 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:12:21,552 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:12:21,554 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:12:21,568 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:12:21,572 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:12:21,575 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:12:21,603 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:12:21,611 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:12:21,616 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:12:21,621 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:12:21,624 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:12:22,168 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:12:22,414 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
notificationworker stdout | 2024-04-22 04:12:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:32.430973+00:00 (in 9.999606 seconds)
notificationworker stdout | 2024-04-22 04:12:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:32 UTC)" (scheduled at 2024-04-22 04:12:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:12:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:12:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 22, 431639), True, datetime.datetime(2024, 4, 22, 4, 12, 22, 431639), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,445 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 52, 431639), 1, '1cda6155-9a44-476c-892a-923854cf6175', 35, '31e57500-4b0d-40e6-aade-435a390631b8'])
notificationworker stdout | 2024-04-22 04:12:22,451 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68473", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0215", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:22,451 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:22,452 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,467 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
builder stdout | 2024-04-22 04:12:22,496 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
notificationworker stdout | 2024-04-22 04:12:22,880 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:22,882 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,887 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,891 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,895 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:22,901 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [35])
notificationworker stdout | 2024-04-22 04:12:22,908 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 22, 908284), True, datetime.datetime(2024, 4, 22, 4, 12, 22, 908284), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,913 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 52, 908284), 1, 'd1985ced-84de-4c61-810b-fece2adbef24', 7, 'd9a91b8e-d337-485a-8493-346a5c909772'])
notificationworker stdout | 2024-04-22 04:12:22,919 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68517", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3446", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:22,919 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:22,920 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:22,934 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
exportactionlogsworker stdout | 2024-04-22 04:12:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:12:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:53.045554+00:00 (in 29.998177 seconds)
exportactionlogsworker stdout | 2024-04-22 04:12:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:17:23 UTC)" (scheduled at 2024-04-22 04:12:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 04:12:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 12, 23, 47733), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:12:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 12, 23, 47733), True, datetime.datetime(2024, 4, 22, 4, 12, 23, 47733), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:12:23,066 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 12, 23, 47733), True, datetime.datetime(2024, 4, 22, 4, 12, 23, 47733), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 4, 12, 23, 47733), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:12:23,070 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:12:23,070 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:17:23 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:12:23,349 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:12:23,375 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:12:23,499 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:23,501 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:23,505 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:23,509 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:23,513 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:23,518 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [7])
notificationworker stdout | 2024-04-22 04:12:23,524 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 23, 523950), True, datetime.datetime(2024, 4, 22, 4, 12, 23, 523950), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:23,529 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 12, 53, 523950), 1, '2d52ab15-90d5-4fae-92d7-778a1ef07d47', 38, 'b8974089-4200-4525-a53f-3d6b8fdc06f7'])
notificationworker stdout | 2024-04-22 04:12:23,534 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68470", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-4450", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:23,534 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:23,535 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['821e891b-da3f-4153-b053-c6689eea2a99', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:23,548 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = ("repositorynotification"."number_of_failures" + %s) WHERE ("repositorynotification"."uuid" = %s)', [1, '821e891b-da3f-4153-b053-c6689eea2a99'])
notificationworker stdout | 2024-04-22 04:12:23,553 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:23,553 [77] [ERROR] [workers.worker] Operation raised exception
notificationworker stdout | Traceback (most recent call last):
notificationworker stdout |   File "/quay-registry/workers/worker.py", line 86, in _operation_func
notificationworker stdout |     return operation_func()
notificationworker stdout |   File "/quay-registry/workers/queueworker.py", line 134, in poll_queue
notificationworker stdout |     self.process_queue_item(job_details)
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 45, in process_queue_item
notificationworker stdout |     raise exc
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 41, in process_queue_item
notificationworker stdout |     method_handler.perform(notification, event_handler, job_details)
notificationworker stdout |   File "/quay-registry/notifications/notificationmethod.py", line 179, in perform
notificationworker stdout |     event_handler.get_summary(notification_data["event_data"], notification_data),
notificationworker stdout |   File "/quay-registry/notifications/notificationevent.py", line 250, in get_summary
notificationworker stdout |     len(event_data["tags"]),
notificationworker stdout | KeyError: 'tags'
notificationworker stdout | 2024-04-22 04:12:23,554 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:32 UTC)" executed successfully
quotaregistrysizeworker stdout | 2024-04-22 04:12:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:12:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:24.738365+00:00 (in 59.999470 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:12:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:24 UTC)" (scheduled at 2024-04-22 04:12:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:12:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:12:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:12:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:12:27,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:12:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:42.806778+00:00 (in 14.999506 seconds)
quotatotalworker stdout | 2024-04-22 04:12:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:42 UTC)" (scheduled at 2024-04-22 04:12:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:12:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:12:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:12:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:12:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:12:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:58.130973+00:00 (in 29.999501 seconds)
buildlogsarchiver stdout | 2024-04-22 04:12:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:58 UTC)" (scheduled at 2024-04-22 04:12:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:12:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 12, 28, 131706), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:12:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:12:28,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:12:28,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:12:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:31.933721+00:00 (in 2.002228 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:29 UTC)" (scheduled at 2024-04-22 04:12:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 29, 931921), True, datetime.datetime(2024, 4, 22, 4, 12, 29, 931921), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:12:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:12:30,632 [285] [DEBUG] [app] Starting request: urn:request:1ca32ad6-2516-42c2-b2e0-37891fd29ea1 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:30,632 [282] [DEBUG] [app] Starting request: urn:request:0ecbf0b0-5a6d-4849-bd85-8990494bbb08 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:12:30,636 [281] [DEBUG] [app] Starting request: urn:request:4137efd5-7146-4566-8974-f452ce328dc7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:30,636 [283] [DEBUG] [app] Starting request: urn:request:d06eccb1-6f55-4e40-8076-bf59723ddf77 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:30,636 [281] [DEBUG] [app] Ending request: urn:request:4137efd5-7146-4566-8974-f452ce328dc7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4137efd5-7146-4566-8974-f452ce328dc7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:30,636 [283] [DEBUG] [app] Ending request: urn:request:d06eccb1-6f55-4e40-8076-bf59723ddf77 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d06eccb1-6f55-4e40-8076-bf59723ddf77', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:12:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:12:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:30,639 [286] [DEBUG] [app] Starting request: urn:request:2a39c44e-c46f-4c38-a443-aae79024d77d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:30,639 [286] [DEBUG] [app] Ending request: urn:request:2a39c44e-c46f-4c38-a443-aae79024d77d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2a39c44e-c46f-4c38-a443-aae79024d77d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:30,639 [285] [DEBUG] [app] Starting request: urn:request:008f00e8-bde4-4289-a25e-370ca6580fe4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:30,640 [285] [DEBUG] [app] Ending request: urn:request:008f00e8-bde4-4289-a25e-370ca6580fe4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:008f00e8-bde4-4289-a25e-370ca6580fe4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:12:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:30,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:30,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:30,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:30,660 [282] [DEBUG] [app] Ending request: urn:request:0ecbf0b0-5a6d-4849-bd85-8990494bbb08 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ecbf0b0-5a6d-4849-bd85-8990494bbb08', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:12:30,661 [285] [DEBUG] [app] Ending request: urn:request:1ca32ad6-2516-42c2-b2e0-37891fd29ea1 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1ca32ad6-2516-42c2-b2e0-37891fd29ea1', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:12:30,661 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:12:30,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:12:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:12:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:01.842666+00:00 (in 29.999433 seconds)
gcworker stdout | 2024-04-22 04:12:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:01 UTC)" (scheduled at 2024-04-22 04:12:31.842666+00:00)
gcworker stdout | 2024-04-22 04:12:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549551843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:12:31,858 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:12:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:12:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:12:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:12:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:29.931002+00:00 (in 57.996778 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:12:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:31 UTC)" (scheduled at 2024-04-22 04:12:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:12:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:12:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:12:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:38.433394+00:00 (in 6.001972 seconds)
notificationworker stdout | 2024-04-22 04:12:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:42 UTC)" (scheduled at 2024-04-22 04:12:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:12:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:12:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 32, 431658), True, datetime.datetime(2024, 4, 22, 4, 12, 32, 431658), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,445 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 2, 431658), 1, 'bc03ab14-1123-4fca-8203-22c322a5e64a', 31, '72d80dc4-639d-4e46-8a22-0b174df6c889'])
notificationworker stdout | 2024-04-22 04:12:32,451 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:32,451 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:32,452 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,466 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
notificationworker stdout | 2024-04-22 04:12:32,881 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:32,883 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,887 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,891 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,895 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:32,900 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [31])
notificationworker stdout | 2024-04-22 04:12:32,906 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 32, 906220), True, datetime.datetime(2024, 4, 22, 4, 12, 32, 906220), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,911 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 2, 906220), 1, '50087148-81c8-4b08-99e9-162bf5aed04d', 15, 'dd8aeaf3-5972-4011-8179-1abc7727aac6'])
notificationworker stdout | 2024-04-22 04:12:32,916 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "55d1704d-84c4-48e0-a0be-df13625280c4", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "71141", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2022-37434", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:32,916 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:32,917 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:32,932 [77] [DEBUG] [urllib3.connectionpool] Starting new HTTPS connection (1): webhook.site:443
notificationworker stdout | 2024-04-22 04:12:33,362 [77] [DEBUG] [urllib3.connectionpool] https://webhook.site:443 "POST /ffd85e96-d5d9-4a80-a40b-342ae3308119 HTTP/1.1" 200 None
notificationworker stdout | 2024-04-22 04:12:33,364 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures" FROM "repositorynotification" AS "t1" WHERE ("t1"."uuid" = %s) LIMIT %s OFFSET %s', ['55d1704d-84c4-48e0-a0be-df13625280c4', 1, 0])
notificationworker stdout | 2024-04-22 04:12:33,368 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_user_id", "t1"."name", "t1"."visibility_id", "t1"."description", "t1"."badge_token", "t1"."kind_id", "t1"."trust_enabled", "t1"."state" FROM "repository" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [1, 1, 0])
notificationworker stdout | 2024-04-22 04:12:33,372 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ("t1"."id" = %s) LIMIT %s OFFSET %s', [2, 1, 0])
notificationworker stdout | 2024-04-22 04:12:33,376 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = %s WHERE ("repositorynotification"."id" = %s)', [0, 7])
notificationworker stdout | 2024-04-22 04:12:33,387 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [15])
notificationworker stdout | 2024-04-22 04:12:33,393 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 33, 392508), True, datetime.datetime(2024, 4, 22, 4, 12, 33, 392508), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:33,397 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 3, 392508), 1, 'ef1c99e8-bc67-487c-b5df-54b4170a2971', 34, 'b6af4c29-b403-4efa-9523-86a5e8fd0ddc'])
notificationworker stdout | 2024-04-22 04:12:33,403 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "821e891b-da3f-4153-b053-c6689eea2a99", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68473", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0215", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:33,403 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:33,404 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['821e891b-da3f-4153-b053-c6689eea2a99', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:33,417 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = ("repositorynotification"."number_of_failures" + %s) WHERE ("repositorynotification"."uuid" = %s)', [1, '821e891b-da3f-4153-b053-c6689eea2a99'])
notificationworker stdout | 2024-04-22 04:12:33,422 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:33,422 [77] [ERROR] [workers.worker] Operation raised exception
notificationworker stdout | Traceback (most recent call last):
notificationworker stdout |   File "/quay-registry/workers/worker.py", line 86, in _operation_func
notificationworker stdout |     return operation_func()
notificationworker stdout |   File "/quay-registry/workers/queueworker.py", line 134, in poll_queue
notificationworker stdout |     self.process_queue_item(job_details)
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 45, in process_queue_item
notificationworker stdout |     raise exc
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 41, in process_queue_item
notificationworker stdout |     method_handler.perform(notification, event_handler, job_details)
notificationworker stdout |   File "/quay-registry/notifications/notificationmethod.py", line 179, in perform
notificationworker stdout |     event_handler.get_summary(notification_data["event_data"], notification_data),
notificationworker stdout |   File "/quay-registry/notifications/notificationevent.py", line 250, in get_summary
notificationworker stdout |     len(event_data["tags"]),
notificationworker stdout | KeyError: 'tags'
notificationworker stdout | 2024-04-22 04:12:33,422 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:12:35,641 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:12:36,921 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
notificationworker stdout | 2024-04-22 04:12:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:42.430973+00:00 (in 3.997092 seconds)
notificationworker stdout | 2024-04-22 04:12:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:17:38 UTC)" (scheduled at 2024-04-22 04:12:38.433394+00:00)
notificationworker stdout | 2024-04-22 04:12:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 12, 38, 434116), 'notification/%'])
notificationworker stdout | 2024-04-22 04:12:38,447 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 12, 38, 434116), True, datetime.datetime(2024, 4, 22, 4, 12, 38, 434116), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 04:12:38,451 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 12, 38, 434116), True, datetime.datetime(2024, 4, 22, 4, 12, 38, 434116), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 4, 12, 38, 434116), 'notification/%'])
notificationworker stdout | 2024-04-22 04:12:38,455 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:38,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:17:38 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:12:39,473 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:12:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:12:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:10.238173+00:00 (in 29.999445 seconds)
autopruneworker stdout | 2024-04-22 04:12:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:10 UTC)" (scheduled at 2024-04-22 04:12:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:12:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759100247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:12:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:12:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:12:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:12:40,455 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:12:40,791 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:12:41,005 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:12:41,485 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:12:42,125 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:12:42,404 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:12:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:52.430973+00:00 (in 9.999515 seconds)
notificationworker stdout | 2024-04-22 04:12:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:52 UTC)" (scheduled at 2024-04-22 04:12:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:12:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:12:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 42, 431863), True, datetime.datetime(2024, 4, 22, 4, 12, 42, 431863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:42,446 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 12, 431863), 1, '21c79a30-dfb2-4cec-9e4e-a0bd1d9fbd18', 24, '5d6e6c78-6000-412f-9ef2-f5701a4b33f4'])
notificationworker stdout | 2024-04-22 04:12:42,451 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68526", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0465", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:42,451 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:42,452 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['27d600e3-69ac-4c25-9b0d-4999e865fe2c', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:42,466 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE (("t1"."username" = %s) AND ("t1"."robot" = %s)) LIMIT %s OFFSET %s', ['build_test', False, 1, 0])
notificationworker stdout | 2024-04-22 04:12:42,470 [77] [DEBUG] [peewee] ('UPDATE "repositorynotification" SET "number_of_failures" = ("repositorynotification"."number_of_failures" + %s) WHERE ("repositorynotification"."uuid" = %s)', [1, '27d600e3-69ac-4c25-9b0d-4999e865fe2c'])
notificationworker stdout | 2024-04-22 04:12:42,475 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:42,475 [77] [ERROR] [workers.worker] Operation raised exception
notificationworker stdout | Traceback (most recent call last):
notificationworker stdout |   File "/quay-registry/workers/worker.py", line 86, in _operation_func
notificationworker stdout |     return operation_func()
notificationworker stdout |   File "/quay-registry/workers/queueworker.py", line 134, in poll_queue
notificationworker stdout |     self.process_queue_item(job_details)
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 45, in process_queue_item
notificationworker stdout |     raise exc
notificationworker stdout |   File "/quay-registry/workers/notificationworker/notificationworker.py", line 41, in process_queue_item
notificationworker stdout |     method_handler.perform(notification, event_handler, job_details)
notificationworker stdout |   File "/quay-registry/notifications/notificationmethod.py", line 491, in perform
notificationworker stdout |     summary = event_handler.get_summary(notification_data["event_data"], notification_data)
notificationworker stdout |   File "/quay-registry/notifications/notificationevent.py", line 250, in get_summary
notificationworker stdout |     len(event_data["tags"]),
notificationworker stdout | KeyError: 'tags'
notificationworker stdout | 2024-04-22 04:12:42,476 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:12:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:12:42,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:12:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:57.806778+00:00 (in 14.999491 seconds)
quotatotalworker stdout | 2024-04-22 04:12:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:57 UTC)" (scheduled at 2024-04-22 04:12:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:12:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:12:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:12:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:12:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:12:43,657 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:12:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:12:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:47.065745+00:00 (in 2.998166 seconds)
securityworker stdout | 2024-04-22 04:12:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:14 UTC)" (scheduled at 2024-04-22 04:12:44.067050+00:00)
securityworker stdout | 2024-04-22 04:12:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:12:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:12:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:12:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:12:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:12:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:12:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:12:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 7, 44, 77582), 1, 2])
securityworker stdout | 2024-04-22 04:12:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 7, 44, 77582), 1, 2])
securityworker stdout | 2024-04-22 04:12:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:12:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:12:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:12:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:12:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:12:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:12:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:12:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:12:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:12:44,675 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:12:45,632 [286] [DEBUG] [app] Starting request: urn:request:bb56ca1b-5a4f-4a81-bb8e-6345075fd7d7 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:45,632 [285] [DEBUG] [app] Starting request: urn:request:b71e2bf9-50a2-4b12-b9c5-e0a316fb2fba (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:12:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:12:45,635 [279] [DEBUG] [app] Starting request: urn:request:48ddf8f9-ffc3-4de7-94c5-951043958601 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:45,636 [278] [DEBUG] [app] Starting request: urn:request:451dd3e9-9b20-4bdf-9b5d-0a163a41fbe0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:12:45,636 [279] [DEBUG] [app] Ending request: urn:request:48ddf8f9-ffc3-4de7-94c5-951043958601 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:48ddf8f9-ffc3-4de7-94c5-951043958601', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:45,636 [278] [DEBUG] [app] Ending request: urn:request:451dd3e9-9b20-4bdf-9b5d-0a163a41fbe0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:451dd3e9-9b20-4bdf-9b5d-0a163a41fbe0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:12:45,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:12:45,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:12:45,638 [285] [DEBUG] [app] Starting request: urn:request:126033fd-e340-4a4c-b5ce-97f08d667e37 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:12:45,639 [285] [DEBUG] [app] Ending request: urn:request:126033fd-e340-4a4c-b5ce-97f08d667e37 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:126033fd-e340-4a4c-b5ce-97f08d667e37', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:12:45,639 [282] [DEBUG] [app] Starting request: urn:request:8f160948-a39e-4f17-8f19-b77e699146a0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:12:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:45,639 [282] [DEBUG] [app] Ending request: urn:request:8f160948-a39e-4f17-8f19-b77e699146a0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8f160948-a39e-4f17-8f19-b77e699146a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:12:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:12:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:12:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:12:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:12:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:12:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:12:45,659 [285] [DEBUG] [app] Ending request: urn:request:b71e2bf9-50a2-4b12-b9c5-e0a316fb2fba (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b71e2bf9-50a2-4b12-b9c5-e0a316fb2fba', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:45,659 [286] [DEBUG] [app] Ending request: urn:request:bb56ca1b-5a4f-4a81-bb8e-6345075fd7d7 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bb56ca1b-5a4f-4a81-bb8e-6345075fd7d7', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:12:45,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:12:45,659 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:12:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:12:45,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:12:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:12:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:12:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:12:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:14.067050+00:00 (in 27.000697 seconds)
securityworker stdout | 2024-04-22 04:12:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:17 UTC)" (scheduled at 2024-04-22 04:12:47.065745+00:00)
securityworker stdout | 2024-04-22 04:12:47,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:12:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:12:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:12:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:12:47,083 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:12:47,083 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:17 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:12:47,192 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
namespacegcworker stdout | 2024-04-22 04:12:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:12:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:10.248668+00:00 (in 23.002320 seconds)
namespacegcworker stdout | 2024-04-22 04:12:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:47 UTC)" (scheduled at 2024-04-22 04:12:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:12:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:12:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 47, 246629), True, datetime.datetime(2024, 4, 22, 4, 12, 47, 246629), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:12:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:12:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:12:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:12:47,319 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:12:47,859 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:12:48,478 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:12:49,428 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:12:49,956 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:12:49,959 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:12:49,961 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:12:50,524 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:12:50,936 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:12:51,490 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:12:51,553 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:12:51,555 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:12:51,559 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:12:51,562 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:12:51,564 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:12:51,577 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:12:51,581 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:12:51,584 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:12:51,610 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:12:51,617 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:12:51,623 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:12:51,628 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:12:51,631 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:12:52,194 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:12:52,429 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
notificationworker stdout | 2024-04-22 04:12:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:12:54.433683+00:00 (in 2.002293 seconds)
notificationworker stdout | 2024-04-22 04:12:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:02 UTC)" (scheduled at 2024-04-22 04:12:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:12:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:12:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 52, 431616), True, datetime.datetime(2024, 4, 22, 4, 12, 52, 431616), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:52,445 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 22, 431616), 1, '450ee019-a8ff-4b16-98e5-bba3e0eb8fb6', 32, '6e173ba1-6434-4502-a331-7da1d0836faa'])
notificationworker stdout | 2024-04-22 04:12:52,451 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68520", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-5678", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:12:52,451 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:52,452 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['27d600e3-69ac-4c25-9b0d-4999e865fe2c', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:12:52,466 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [32])
notificationworker stdout | 2024-04-22 04:12:52,472 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 52, 471694), True, datetime.datetime(2024, 4, 22, 4, 12, 52, 471694), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:12:52,476 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:12:52,476 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:12:52,477 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:02 UTC)" executed successfully
builder stdout | 2024-04-22 04:12:52,522 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:12:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:12:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:18.047111+00:00 (in 25.001089 seconds)
exportactionlogsworker stdout | 2024-04-22 04:12:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:53 UTC)" (scheduled at 2024-04-22 04:12:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:12:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:12:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 12, 53, 46320), True, datetime.datetime(2024, 4, 22, 4, 12, 53, 46320), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:12:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:12:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:12:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:12:53,367 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:12:53,382 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:12:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:12:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:02.430973+00:00 (in 7.996786 seconds)
notificationworker stdout | 2024-04-22 04:12:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:54 UTC)" (scheduled at 2024-04-22 04:12:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:12:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:12:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:13:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:12:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:12:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:12.806778+00:00 (in 14.999411 seconds)
quotatotalworker stdout | 2024-04-22 04:12:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:12 UTC)" (scheduled at 2024-04-22 04:12:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:12:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:12:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:12:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:12:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:12:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:28.130973+00:00 (in 29.999552 seconds)
buildlogsarchiver stdout | 2024-04-22 04:12:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:28 UTC)" (scheduled at 2024-04-22 04:12:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:12:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 12, 58, 131620), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:12:58,144 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:12:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:12:58,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:13:00,632 [282] [DEBUG] [app] Starting request: urn:request:cd7c6129-b747-4ab4-89e9-028766a4ac47 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:00,632 [285] [DEBUG] [app] Starting request: urn:request:de2cee15-ab9c-480d-b65d-97f41bc03ff8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:13:00,636 [278] [DEBUG] [app] Starting request: urn:request:34043e4c-83e0-49a5-b151-12f45485f882 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:00,636 [281] [DEBUG] [app] Starting request: urn:request:9b846345-fb06-4752-a819-89c65a0dc790 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:00,636 [278] [DEBUG] [app] Ending request: urn:request:34043e4c-83e0-49a5-b151-12f45485f882 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:34043e4c-83e0-49a5-b151-12f45485f882', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:00,636 [281] [DEBUG] [app] Ending request: urn:request:9b846345-fb06-4752-a819-89c65a0dc790 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9b846345-fb06-4752-a819-89c65a0dc790', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:00,637 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:13:00,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:00,639 [282] [DEBUG] [app] Starting request: urn:request:f12d634d-43ac-4643-b19e-76ea2cbf8395 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:00,639 [282] [DEBUG] [app] Ending request: urn:request:f12d634d-43ac-4643-b19e-76ea2cbf8395 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f12d634d-43ac-4643-b19e-76ea2cbf8395', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:00,640 [284] [DEBUG] [app] Starting request: urn:request:f9693a1e-6139-4894-b898-415c429a7628 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:00,640 [284] [DEBUG] [app] Ending request: urn:request:f9693a1e-6139-4894-b898-415c429a7628 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f9693a1e-6139-4894-b898-415c429a7628', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:00,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:00,660 [282] [DEBUG] [app] Ending request: urn:request:cd7c6129-b747-4ab4-89e9-028766a4ac47 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cd7c6129-b747-4ab4-89e9-028766a4ac47', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:00,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:13:00,660 [285] [DEBUG] [app] Ending request: urn:request:de2cee15-ab9c-480d-b65d-97f41bc03ff8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:de2cee15-ab9c-480d-b65d-97f41bc03ff8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:13:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:00,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:13:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:13:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:31.842666+00:00 (in 29.999446 seconds)
gcworker stdout | 2024-04-22 04:13:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:31 UTC)" (scheduled at 2024-04-22 04:13:01.842666+00:00)
gcworker stdout | 2024-04-22 04:13:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549581843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:13:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:13:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:13:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:13:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:13:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:09.033206+00:00 (in 6.997034 seconds)
repositorygcworker stdout | 2024-04-22 04:13:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:02 UTC)" (scheduled at 2024-04-22 04:13:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:13:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:13:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:13:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:12.430973+00:00 (in 9.999531 seconds)
notificationworker stdout | 2024-04-22 04:13:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:12 UTC)" (scheduled at 2024-04-22 04:13:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:13:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:13:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 2, 431672), True, datetime.datetime(2024, 4, 22, 4, 13, 2, 431672), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:02,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:13:02,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:13:05,678 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:13:06,929 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:13:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:13:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:26.035364+00:00 (in 17.001680 seconds)
repositorygcworker stdout | 2024-04-22 04:13:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:09 UTC)" (scheduled at 2024-04-22 04:13:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:13:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:13:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 9, 34119), True, datetime.datetime(2024, 4, 22, 4, 13, 9, 34119), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:13:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:13:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:13:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:13:09,493 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:13:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:13:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:40.238173+00:00 (in 29.999464 seconds)
autopruneworker stdout | 2024-04-22 04:13:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:40 UTC)" (scheduled at 2024-04-22 04:13:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:13:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759130247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:13:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:13:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:47.245913+00:00 (in 36.996764 seconds)
namespacegcworker stdout | 2024-04-22 04:13:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:10 UTC)" (scheduled at 2024-04-22 04:13:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:13:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:13:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:13:10,252 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:13:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:13:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:13:10,479 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:13:10,804 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:13:11,041 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:13:11,501 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:13:12,147 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:13:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:22.430973+00:00 (in 9.999495 seconds)
notificationworker stdout | 2024-04-22 04:13:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:22 UTC)" (scheduled at 2024-04-22 04:13:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:13:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:13:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 12, 431921), True, datetime.datetime(2024, 4, 22, 4, 13, 12, 431921), 0, 'notification/%', 50, 1, 0])
storagereplication stdout | 2024-04-22 04:13:12,440 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
notificationworker stdout | 2024-04-22 04:13:12,446 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 42, 431921), 1, '01883fc3-b467-4d82-a7a5-3599c9059dfa', 48, '5bfeab3f-af03-497f-b57a-f4a19ad3e057'])
notificationworker stdout | 2024-04-22 04:13:12,452 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68450", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-3817", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:13:12,452 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:12,453 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['27d600e3-69ac-4c25-9b0d-4999e865fe2c', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:13:12,467 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [48])
notificationworker stdout | 2024-04-22 04:13:12,473 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 12, 472507), True, datetime.datetime(2024, 4, 22, 4, 13, 12, 472507), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:12,477 [77] [DEBUG] [peewee] ('UPDATE "queueitem" SET "available" = %s, "processing_expires" = %s, "retries_remaining" = ("queueitem"."retries_remaining" - %s), "state_id" = %s WHERE (("queueitem"."id" = %s) AND ("queueitem"."state_id" = %s))', [False, datetime.datetime(2024, 4, 22, 4, 13, 42, 472507), 1, 'ce51d13f-5612-477b-a2b1-f6b40386d825', 24, '21c79a30-dfb2-4cec-9e4e-a0bd1d9fbd18'])
notificationworker stdout | 2024-04-22 04:13:12,482 [77] [DEBUG] [workers.queueworker] Queue gave us some work: {"notification_uuid": "27d600e3-69ac-4c25-9b0d-4999e865fe2c", "event_data": {"repository": "build_test/pushno", "namespace": "build_test", "name": "pushno", "docker_url": "quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/build_test/pushno", "homepage": "https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com/repository/build_test/pushno", "vulnerable_index_report_created": "true", "vulnerability": {"id": "68526", "description": "", "link": "https://www.cve.org/CVERecord?id=CVE-2023-0465", "priority": "", "has_fix": true}}, "performer_data": {}}
notificationworker stdout | 2024-04-22 04:13:12,482 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:12,484 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."repository_id", "t1"."event_id", "t1"."method_id", "t1"."title", "t1"."config_json", "t1"."event_config_json", "t1"."number_of_failures", "t2"."id", "t2"."namespace_user_id", "t2"."name", "t2"."visibility_id", "t2"."description", "t2"."badge_token", "t2"."kind_id", "t2"."trust_enabled", "t2"."state", "t3"."id", "t3"."uuid", "t3"."username", "t3"."password_hash", "t3"."email", "t3"."verified", "t3"."stripe_id", "t3"."organization", "t3"."robot", "t3"."invoice_email", "t3"."invalid_login_attempts", "t3"."last_invalid_login", "t3"."removed_tag_expiration_s", "t3"."enabled", "t3"."invoice_email_address", "t3"."given_name", "t3"."family_name", "t3"."company", "t3"."location", "t3"."maximum_queued_builds_count", "t3"."creation_date", "t3"."last_accessed" FROM "repositorynotification" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE (("t1"."uuid" = %s) AND ("t1"."number_of_failures" < %s)) LIMIT %s OFFSET %s', ['27d600e3-69ac-4c25-9b0d-4999e865fe2c', 3, 1, 0])
notificationworker stdout | 2024-04-22 04:13:12,497 [77] [DEBUG] [peewee] ('DELETE FROM "queueitem" WHERE ("queueitem"."id" = %s)', [24])
notificationworker stdout | 2024-04-22 04:13:12,503 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 12, 502721), True, datetime.datetime(2024, 4, 22, 4, 13, 12, 502721), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:12,507 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:13:12,507 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:12,507 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:13:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:13:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:27.806778+00:00 (in 14.999471 seconds)
quotatotalworker stdout | 2024-04-22 04:13:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:27 UTC)" (scheduled at 2024-04-22 04:13:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:13:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:13:12,820 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:13:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:13:13,694 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:13:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:13:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:17.065745+00:00 (in 2.998127 seconds)
securityworker stdout | 2024-04-22 04:13:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:44 UTC)" (scheduled at 2024-04-22 04:13:14.067050+00:00)
securityworker stdout | 2024-04-22 04:13:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:13:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:13:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:13:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:13:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:13:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:13:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:13:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:13:14,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 8, 14, 77222), 1, 2])
securityworker stdout | 2024-04-22 04:13:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 8, 14, 77222), 1, 2])
securityworker stdout | 2024-04-22 04:13:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:14,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:14,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:13:14,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:13:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:13:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:13:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:13:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:13:14,700 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:13:15,632 [285] [DEBUG] [app] Starting request: urn:request:da2b4d1a-e696-47c0-8768-87b3f8e4780a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:15,633 [282] [DEBUG] [app] Starting request: urn:request:ddb187ae-6cf4-466a-a6a9-2030b77e1ae9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:13:15,636 [278] [DEBUG] [app] Starting request: urn:request:39e7266d-697c-47c7-a15c-46d5582e4592 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:15,636 [275] [DEBUG] [app] Starting request: urn:request:b8342c41-2d2f-46e0-aa92-0eef1d007530 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:15,636 [278] [DEBUG] [app] Ending request: urn:request:39e7266d-697c-47c7-a15c-46d5582e4592 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:39e7266d-697c-47c7-a15c-46d5582e4592', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:15,636 [275] [DEBUG] [app] Ending request: urn:request:b8342c41-2d2f-46e0-aa92-0eef1d007530 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b8342c41-2d2f-46e0-aa92-0eef1d007530', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:15,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:13:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:13:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:15,639 [285] [DEBUG] [app] Starting request: urn:request:c7e5b474-0c04-48d1-89c2-1c25fffd9ab7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:15,639 [285] [DEBUG] [app] Ending request: urn:request:c7e5b474-0c04-48d1-89c2-1c25fffd9ab7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c7e5b474-0c04-48d1-89c2-1c25fffd9ab7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:15,639 [286] [DEBUG] [app] Starting request: urn:request:12cc0886-41f9-4eb4-a577-1f3cf1097f70 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:15,639 [286] [DEBUG] [app] Ending request: urn:request:12cc0886-41f9-4eb4-a577-1f3cf1097f70 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:12cc0886-41f9-4eb4-a577-1f3cf1097f70', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:15,659 [285] [DEBUG] [app] Ending request: urn:request:da2b4d1a-e696-47c0-8768-87b3f8e4780a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:da2b4d1a-e696-47c0-8768-87b3f8e4780a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:15,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:13:15,660 [282] [DEBUG] [app] Ending request: urn:request:ddb187ae-6cf4-466a-a6a9-2030b77e1ae9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ddb187ae-6cf4-466a-a6a9-2030b77e1ae9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
securityworker stdout | 2024-04-22 04:13:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:13:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:44.067050+00:00 (in 27.000745 seconds)
securityworker stdout | 2024-04-22 04:13:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:47 UTC)" (scheduled at 2024-04-22 04:13:17.065745+00:00)
securityworker stdout | 2024-04-22 04:13:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:13:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:13:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:13:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:13:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:13:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:13:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 8, 17, 70705), 1, 2])
securityworker stdout | 2024-04-22 04:13:17,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 8, 17, 70705), 1, 2])
securityworker stdout | 2024-04-22 04:13:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:17,099 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:13:17,099 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:13:17,229 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:13:17,355 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:13:17,882 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:13:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:13:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:53.045554+00:00 (in 34.997809 seconds)
exportactionlogsworker stdout | 2024-04-22 04:13:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:18 UTC)" (scheduled at 2024-04-22 04:13:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:13:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:13:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:13:18,507 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:13:19,441 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:13:19,966 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:13:19,969 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:13:19,971 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:13:20,549 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:13:20,967 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:13:21,500 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:13:21,564 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:13:21,566 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:13:21,568 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:13:21,571 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:13:21,575 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:13:21,584 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:13:21,588 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:13:21,591 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:13:21,618 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:13:21,624 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:13:21,629 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:13:21,636 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:13:21,638 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:13:22,231 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:13:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:32.430973+00:00 (in 9.999480 seconds)
notificationworker stdout | 2024-04-22 04:13:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:32 UTC)" (scheduled at 2024-04-22 04:13:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:13:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:13:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 22, 431933), True, datetime.datetime(2024, 4, 22, 4, 13, 22, 431933), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:13:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:32 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:13:22,465 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:13:22,548 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
quotatotalworker stdout | 2024-04-22 04:13:23,377 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:13:23,404 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:13:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:13:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:24.738365+00:00 (in 59.999472 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:13:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:24 UTC)" (scheduled at 2024-04-22 04:13:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:13:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:13:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:13:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:13:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:13:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:02.035613+00:00 (in 35.999804 seconds)
repositorygcworker stdout | 2024-04-22 04:13:26,035 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:18:26 UTC)" (scheduled at 2024-04-22 04:13:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 04:13:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 13, 26, 36027), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:13:26,049 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 13, 26, 36027), True, datetime.datetime(2024, 4, 22, 4, 13, 26, 36027), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:13:26,054 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 13, 26, 36027), True, datetime.datetime(2024, 4, 22, 4, 13, 26, 36027), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 4, 13, 26, 36027), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:13:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:13:26,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:18:26 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:13:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:13:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:42.806778+00:00 (in 14.999409 seconds)
quotatotalworker stdout | 2024-04-22 04:13:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:42 UTC)" (scheduled at 2024-04-22 04:13:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:13:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:13:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:13:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:13:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:13:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:58.130973+00:00 (in 29.999481 seconds)
buildlogsarchiver stdout | 2024-04-22 04:13:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:58 UTC)" (scheduled at 2024-04-22 04:13:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:13:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 13, 28, 131722), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:13:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:13:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:13:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:13:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:30.933445+00:00 (in 1.001932 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:29 UTC)" (scheduled at 2024-04-22 04:13:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 29, 931814), True, datetime.datetime(2024, 4, 22, 4, 13, 29, 931814), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,945 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,945 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:13:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:13:30,632 [282] [DEBUG] [app] Starting request: urn:request:78d098a5-33d9-4555-bd58-66e267668b23 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:30,632 [285] [DEBUG] [app] Starting request: urn:request:65d46127-26aa-4d52-946e-74a21e7abfa8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:13:30,635 [283] [DEBUG] [app] Starting request: urn:request:1bd5969f-c1aa-4a39-83f7-1ce6b88378ba (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:30,635 [276] [DEBUG] [app] Starting request: urn:request:086b23c8-496a-4ca9-b53f-3251f9eb6f34 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:30,636 [283] [DEBUG] [app] Ending request: urn:request:1bd5969f-c1aa-4a39-83f7-1ce6b88378ba (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1bd5969f-c1aa-4a39-83f7-1ce6b88378ba', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:30,636 [276] [DEBUG] [app] Ending request: urn:request:086b23c8-496a-4ca9-b53f-3251f9eb6f34 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:086b23c8-496a-4ca9-b53f-3251f9eb6f34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:30,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:13:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:13:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:30,639 [285] [DEBUG] [app] Starting request: urn:request:5ad4a36d-86d6-41a9-9fff-b6bc69665ae4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:30,639 [285] [DEBUG] [app] Ending request: urn:request:5ad4a36d-86d6-41a9-9fff-b6bc69665ae4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ad4a36d-86d6-41a9-9fff-b6bc69665ae4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:30,639 [286] [DEBUG] [app] Starting request: urn:request:febd6acb-d8dc-4bbf-a69d-8e80ec2c85a0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:30,640 [286] [DEBUG] [app] Ending request: urn:request:febd6acb-d8dc-4bbf-a69d-8e80ec2c85a0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:febd6acb-d8dc-4bbf-a69d-8e80ec2c85a0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:30,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:30,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:30,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:30,660 [285] [DEBUG] [app] Ending request: urn:request:65d46127-26aa-4d52-946e-74a21e7abfa8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:65d46127-26aa-4d52-946e-74a21e7abfa8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:30,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:30,660 [282] [DEBUG] [app] Ending request: urn:request:78d098a5-33d9-4555-bd58-66e267668b23 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:78d098a5-33d9-4555-bd58-66e267668b23', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:30,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.028)
gunicorn-web stdout | 2024-04-22 04:13:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:31.933721+00:00 (in 0.999811 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:18:30 UTC)" (scheduled at 2024-04-22 04:13:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 13, 30, 934151), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,947 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 13, 30, 934151), True, datetime.datetime(2024, 4, 22, 4, 13, 30, 934151), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,951 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 13, 30, 934151), True, datetime.datetime(2024, 4, 22, 4, 13, 30, 934151), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 4, 13, 30, 934151), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,955 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:13:30,955 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:18:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:13:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:13:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:01.842666+00:00 (in 29.999490 seconds)
gcworker stdout | 2024-04-22 04:13:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:01 UTC)" (scheduled at 2024-04-22 04:13:31.842666+00:00)
gcworker stdout | 2024-04-22 04:13:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549611843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:13:31,858 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:13:31,858 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:13:31,858 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:13:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:13:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:29.931002+00:00 (in 57.996797 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:13:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:31 UTC)" (scheduled at 2024-04-22 04:13:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:13:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:13:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:13:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:42.430973+00:00 (in 9.999476 seconds)
notificationworker stdout | 2024-04-22 04:13:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:42 UTC)" (scheduled at 2024-04-22 04:13:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:13:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:13:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 32, 431909), True, datetime.datetime(2024, 4, 22, 4, 13, 32, 431909), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:13:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:13:35,692 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:13:36,967 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:13:39,500 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:13:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:13:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:10.238173+00:00 (in 29.999473 seconds)
autopruneworker stdout | 2024-04-22 04:13:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:10 UTC)" (scheduled at 2024-04-22 04:13:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:13:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759160247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:13:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:13:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:13:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:13:40,486 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:13:40,813 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:13:41,077 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:13:41,523 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:13:42,184 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:13:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:52.430973+00:00 (in 9.999508 seconds)
notificationworker stdout | 2024-04-22 04:13:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:52 UTC)" (scheduled at 2024-04-22 04:13:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:13:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:13:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 42, 431884), True, datetime.datetime(2024, 4, 22, 4, 13, 42, 431884), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:13:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:13:52 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:13:42,477 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:13:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:13:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:57.806778+00:00 (in 14.999465 seconds)
quotatotalworker stdout | 2024-04-22 04:13:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:57 UTC)" (scheduled at 2024-04-22 04:13:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:13:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:13:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:13:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:13:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:13:43,730 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:13:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:13:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:47.065745+00:00 (in 2.998168 seconds)
securityworker stdout | 2024-04-22 04:13:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:14 UTC)" (scheduled at 2024-04-22 04:13:44.067050+00:00)
securityworker stdout | 2024-04-22 04:13:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:13:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:13:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:13:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:13:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:13:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:13:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:13:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:13:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:44,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 8, 44, 78661), 1, 2])
securityworker stdout | 2024-04-22 04:13:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:44,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 8, 44, 78661), 1, 2])
securityworker stdout | 2024-04-22 04:13:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:13:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:13:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:13:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:13:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:13:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:13:44,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:13:44,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:13:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:13:44,720 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:13:45,632 [286] [DEBUG] [app] Starting request: urn:request:cf593c1b-a241-4b98-9874-c0c9e1832816 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:45,632 [285] [DEBUG] [app] Starting request: urn:request:3878c0cd-afeb-441f-8b77-73685c82edaf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:13:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:13:45,635 [283] [DEBUG] [app] Starting request: urn:request:67ae4b98-4464-464a-85a4-a234cae29f7a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:45,635 [283] [DEBUG] [app] Ending request: urn:request:67ae4b98-4464-464a-85a4-a234cae29f7a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:67ae4b98-4464-464a-85a4-a234cae29f7a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:13:45,635 [279] [DEBUG] [app] Starting request: urn:request:ab29136d-b378-49a6-a597-06c5ccb67c4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:13:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:13:45,636 [279] [DEBUG] [app] Ending request: urn:request:ab29136d-b378-49a6-a597-06c5ccb67c4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ab29136d-b378-49a6-a597-06c5ccb67c4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:13:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:13:45,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:13:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:13:45,638 [285] [DEBUG] [app] Starting request: urn:request:c02eab93-4ba5-4550-a16d-88a5763dd9f3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:45,638 [285] [DEBUG] [app] Ending request: urn:request:c02eab93-4ba5-4550-a16d-88a5763dd9f3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c02eab93-4ba5-4550-a16d-88a5763dd9f3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:45,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:45,640 [282] [DEBUG] [app] Starting request: urn:request:23defbc9-0531-4cad-be42-3e65c1dd2f7c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:13:45,640 [282] [DEBUG] [app] Ending request: urn:request:23defbc9-0531-4cad-be42-3e65c1dd2f7c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:23defbc9-0531-4cad-be42-3e65c1dd2f7c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:13:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:13:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:13:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:13:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:13:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:13:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:13:45,659 [285] [DEBUG] [app] Ending request: urn:request:3878c0cd-afeb-441f-8b77-73685c82edaf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3878c0cd-afeb-441f-8b77-73685c82edaf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:45,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:45,659 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:13:45,660 [286] [DEBUG] [app] Ending request: urn:request:cf593c1b-a241-4b98-9874-c0c9e1832816 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cf593c1b-a241-4b98-9874-c0c9e1832816', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:13:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:13:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:13:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:13:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:13:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:13:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:14.067050+00:00 (in 27.000791 seconds)
securityworker stdout | 2024-04-22 04:13:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:17 UTC)" (scheduled at 2024-04-22 04:13:47.065745+00:00)
securityworker stdout | 2024-04-22 04:13:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:13:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:13:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:13:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:13:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:13:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:13:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
expiredappspecifictokenworker stdout | 2024-04-22 04:13:47,246 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
namespacegcworker stdout | 2024-04-22 04:13:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:10.248668+00:00 (in 23.002298 seconds)
namespacegcworker stdout | 2024-04-22 04:13:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:47 UTC)" (scheduled at 2024-04-22 04:13:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:13:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:13:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 47, 246644), True, datetime.datetime(2024, 4, 22, 4, 13, 47, 246644), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:13:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:13:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:13:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:47 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:13:47,393 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:13:47,918 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:13:48,518 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:13:49,477 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:13:49,975 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:13:49,978 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:13:49,980 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:13:50,586 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:13:50,979 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:13:51,507 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:13:51,572 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:13:51,575 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:13:51,577 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:13:51,583 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:13:51,585 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:13:51,591 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:13:51,594 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:13:51,602 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:13:51,624 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:13:51,631 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:13:51,635 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:13:51,644 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:13:51,646 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:13:52,267 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:13:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:13:54.433683+00:00 (in 2.002222 seconds)
notificationworker stdout | 2024-04-22 04:13:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:02 UTC)" (scheduled at 2024-04-22 04:13:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:13:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:13:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 52, 431885), True, datetime.datetime(2024, 4, 22, 4, 13, 52, 431885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:13:52,452 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:13:52,452 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:13:52,452 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:02 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:13:52,501 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:13:52,568 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:13:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:13:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:18.047111+00:00 (in 25.001075 seconds)
exportactionlogsworker stdout | 2024-04-22 04:13:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:53 UTC)" (scheduled at 2024-04-22 04:13:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:13:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:13:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 13, 53, 46316), True, datetime.datetime(2024, 4, 22, 4, 13, 53, 46316), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:13:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:13:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:13:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:13:53,414 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:13:53,428 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:13:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:13:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:02.430973+00:00 (in 7.996837 seconds)
notificationworker stdout | 2024-04-22 04:13:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:54 UTC)" (scheduled at 2024-04-22 04:13:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:13:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:13:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:14:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:13:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:13:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:12.806778+00:00 (in 14.999438 seconds)
quotatotalworker stdout | 2024-04-22 04:13:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:12 UTC)" (scheduled at 2024-04-22 04:13:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:13:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:13:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:13:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:13:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:13:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:28.130973+00:00 (in 29.999538 seconds)
buildlogsarchiver stdout | 2024-04-22 04:13:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:28 UTC)" (scheduled at 2024-04-22 04:13:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:13:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 13, 58, 131643), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:13:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:13:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:13:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:14:00,632 [285] [DEBUG] [app] Starting request: urn:request:eec7da9e-4725-4e30-a686-f831ad735fbf (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:00,632 [286] [DEBUG] [app] Starting request: urn:request:15be0060-c22f-4859-abd8-6bd4143cd0f5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:14:00,636 [275] [DEBUG] [app] Starting request: urn:request:8c7804c6-cbec-4c24-a8a8-ca4168c580de (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:00,636 [281] [DEBUG] [app] Starting request: urn:request:8762c7ed-8dc7-449b-987c-abf61d673ad5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:00,636 [275] [DEBUG] [app] Ending request: urn:request:8c7804c6-cbec-4c24-a8a8-ca4168c580de (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8c7804c6-cbec-4c24-a8a8-ca4168c580de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:14:00,636 [281] [DEBUG] [app] Ending request: urn:request:8762c7ed-8dc7-449b-987c-abf61d673ad5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8762c7ed-8dc7-449b-987c-abf61d673ad5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:14:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:14:00,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:14:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:00,639 [284] [DEBUG] [app] Starting request: urn:request:bd0a6eec-2af7-4263-a322-a1fad8edbfd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:00,640 [284] [DEBUG] [app] Ending request: urn:request:bd0a6eec-2af7-4263-a322-a1fad8edbfd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:bd0a6eec-2af7-4263-a322-a1fad8edbfd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:14:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:00,640 [282] [DEBUG] [app] Starting request: urn:request:138f693a-395d-45aa-a8bc-a2e9c8180b33 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:00,640 [282] [DEBUG] [app] Ending request: urn:request:138f693a-395d-45aa-a8bc-a2e9c8180b33 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:138f693a-395d-45aa-a8bc-a2e9c8180b33', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:00,641 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:00,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:14:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:00,661 [286] [DEBUG] [app] Ending request: urn:request:15be0060-c22f-4859-abd8-6bd4143cd0f5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:15be0060-c22f-4859-abd8-6bd4143cd0f5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:00,661 [285] [DEBUG] [app] Ending request: urn:request:eec7da9e-4725-4e30-a686-f831ad735fbf (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eec7da9e-4725-4e30-a686-f831ad735fbf', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:14:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:14:00,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:14:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:14:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:14:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:31.842666+00:00 (in 29.999472 seconds)
gcworker stdout | 2024-04-22 04:14:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:31 UTC)" (scheduled at 2024-04-22 04:14:01.842666+00:00)
gcworker stdout | 2024-04-22 04:14:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549641843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:14:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:14:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:14:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:14:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:14:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:09.033206+00:00 (in 6.997054 seconds)
repositorygcworker stdout | 2024-04-22 04:14:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:02 UTC)" (scheduled at 2024-04-22 04:14:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:14:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:14:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:14:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:12.430973+00:00 (in 9.999521 seconds)
notificationworker stdout | 2024-04-22 04:14:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:12 UTC)" (scheduled at 2024-04-22 04:14:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:14:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:14:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 2, 431895), True, datetime.datetime(2024, 4, 22, 4, 14, 2, 431895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:14:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:14:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:14:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:14:05,711 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:14:07,002 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:14:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:14:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:02.035613+00:00 (in 53.001909 seconds)
repositorygcworker stdout | 2024-04-22 04:14:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:09 UTC)" (scheduled at 2024-04-22 04:14:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:14:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:14:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 9, 33945), True, datetime.datetime(2024, 4, 22, 4, 14, 9, 33945), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:14:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:14:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:14:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:14:09,537 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:14:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:14:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:40.238173+00:00 (in 29.999513 seconds)
autopruneworker stdout | 2024-04-22 04:14:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:40 UTC)" (scheduled at 2024-04-22 04:14:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:14:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759190247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:14:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:14:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:47.245913+00:00 (in 36.996797 seconds)
namespacegcworker stdout | 2024-04-22 04:14:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:10 UTC)" (scheduled at 2024-04-22 04:14:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:14:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:14:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:14:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:14:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:14:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:14:10,502 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:14:10,828 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:14:11,085 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:14:11,559 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:14:12,193 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:14:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:22.430973+00:00 (in 9.999528 seconds)
notificationworker stdout | 2024-04-22 04:14:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:22 UTC)" (scheduled at 2024-04-22 04:14:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:14:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:14:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 12, 431885), True, datetime.datetime(2024, 4, 22, 4, 14, 12, 431885), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:14:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:14:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:14:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:22 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:14:12,514 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:14:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:14:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:27.806778+00:00 (in 14.999471 seconds)
quotatotalworker stdout | 2024-04-22 04:14:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:27 UTC)" (scheduled at 2024-04-22 04:14:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:14:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:14:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:14:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:14:13,763 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:14:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:14:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:17.065745+00:00 (in 2.998166 seconds)
securityworker stdout | 2024-04-22 04:14:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:44 UTC)" (scheduled at 2024-04-22 04:14:14.067050+00:00)
securityworker stdout | 2024-04-22 04:14:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:14:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:14:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:14:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:14:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:14:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:14:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:14:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:14:14,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:14,093 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 9, 14, 76693), 1, 2])
securityworker stdout | 2024-04-22 04:14:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:14,097 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 9, 14, 76693), 1, 2])
securityworker stdout | 2024-04-22 04:14:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:14,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:14,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:14:14,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:14:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:14:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:14:14,103 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:14,103 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:14:14,103 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:14:14,757 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:14:15,632 [285] [DEBUG] [app] Starting request: urn:request:7dc77cfb-ccb5-4a8f-a002-3ba1c6acb2c3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:15,632 [286] [DEBUG] [app] Starting request: urn:request:a0535da6-4970-4d34-9404-603c285f2e5f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:14:15,635 [281] [DEBUG] [app] Starting request: urn:request:e8d4c996-69d3-45a4-ad7e-85b269b49b02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:15,636 [275] [DEBUG] [app] Starting request: urn:request:5cab45bf-cb43-4dea-a71e-e4b01028ef24 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:15,636 [281] [DEBUG] [app] Ending request: urn:request:e8d4c996-69d3-45a4-ad7e-85b269b49b02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e8d4c996-69d3-45a4-ad7e-85b269b49b02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:14:15,636 [275] [DEBUG] [app] Ending request: urn:request:5cab45bf-cb43-4dea-a71e-e4b01028ef24 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5cab45bf-cb43-4dea-a71e-e4b01028ef24', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:14:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 04:14:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:14:15,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:15,639 [282] [DEBUG] [app] Starting request: urn:request:f520cb9f-8fc7-4906-b034-08e8e613c9ac (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:15,639 [284] [DEBUG] [app] Starting request: urn:request:2fdd4426-b451-4345-a366-ddce05336b35 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:15,640 [282] [DEBUG] [app] Ending request: urn:request:f520cb9f-8fc7-4906-b034-08e8e613c9ac (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f520cb9f-8fc7-4906-b034-08e8e613c9ac', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:15,640 [284] [DEBUG] [app] Ending request: urn:request:2fdd4426-b451-4345-a366-ddce05336b35 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2fdd4426-b451-4345-a366-ddce05336b35', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:14:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:15,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:15,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:15,660 [285] [DEBUG] [app] Ending request: urn:request:7dc77cfb-ccb5-4a8f-a002-3ba1c6acb2c3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7dc77cfb-ccb5-4a8f-a002-3ba1c6acb2c3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:14:15,660 [286] [DEBUG] [app] Ending request: urn:request:a0535da6-4970-4d34-9404-603c285f2e5f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a0535da6-4970-4d34-9404-603c285f2e5f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:14:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 04:14:15,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:14:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:14:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:44.067050+00:00 (in 27.000767 seconds)
securityworker stdout | 2024-04-22 04:14:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:47 UTC)" (scheduled at 2024-04-22 04:14:17.065745+00:00)
securityworker stdout | 2024-04-22 04:14:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:14:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:14:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:14:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:14:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:14:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:14:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 9, 17, 70693), 1, 2])
securityworker stdout | 2024-04-22 04:14:17,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:17,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 9, 17, 70693), 1, 2])
securityworker stdout | 2024-04-22 04:14:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:17,099 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:14:17,099 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:14:17,253 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:14:17,412 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:14:17,936 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:14:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:14:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:53.045554+00:00 (in 34.997854 seconds)
exportactionlogsworker stdout | 2024-04-22 04:14:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:18 UTC)" (scheduled at 2024-04-22 04:14:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:14:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:14:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:14:18,533 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:14:19,501 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:14:19,983 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:14:19,985 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:14:19,988 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:14:20,624 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:14:20,998 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:14:21,520 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:14:21,581 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:14:21,585 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:14:21,587 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:14:21,592 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:14:21,596 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:14:21,599 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:14:21,606 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:14:21,613 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:14:21,631 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:14:21,638 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:14:21,642 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:14:21,650 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:14:21,653 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:14:22,303 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:14:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:32.430973+00:00 (in 9.999547 seconds)
notificationworker stdout | 2024-04-22 04:14:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:32 UTC)" (scheduled at 2024-04-22 04:14:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:14:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:14:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 22, 431808), True, datetime.datetime(2024, 4, 22, 4, 14, 22, 431808), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:14:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:14:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:14:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:32 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:14:22,540 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:14:22,605 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
quotatotalworker stdout | 2024-04-22 04:14:23,450 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:14:23,463 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:14:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:14:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:24.738365+00:00 (in 59.999490 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:14:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:24 UTC)" (scheduled at 2024-04-22 04:14:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:14:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:14:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:14:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:14:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:14:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:42.806778+00:00 (in 14.999442 seconds)
quotatotalworker stdout | 2024-04-22 04:14:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:42 UTC)" (scheduled at 2024-04-22 04:14:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:14:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:14:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:14:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:14:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:14:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:58.130973+00:00 (in 29.999486 seconds)
buildlogsarchiver stdout | 2024-04-22 04:14:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:58 UTC)" (scheduled at 2024-04-22 04:14:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:14:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 14, 28, 131873), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:14:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:14:28,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:14:28,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:14:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:31.933721+00:00 (in 2.002178 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:29 UTC)" (scheduled at 2024-04-22 04:14:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 29, 931964), True, datetime.datetime(2024, 4, 22, 4, 14, 29, 931964), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:14:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:14:30,632 [285] [DEBUG] [app] Starting request: urn:request:993413e9-3381-4287-a5b6-9523ebfe98d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:30,632 [286] [DEBUG] [app] Starting request: urn:request:f56b004e-7173-4261-81ed-bcabcfa69492 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:14:30,635 [275] [DEBUG] [app] Starting request: urn:request:62a73ab0-61aa-4096-a246-6a87fe6578f0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:30,635 [281] [DEBUG] [app] Starting request: urn:request:3c91d882-6073-4d25-b438-e10c5fcca92c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:30,635 [281] [DEBUG] [app] Ending request: urn:request:3c91d882-6073-4d25-b438-e10c5fcca92c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3c91d882-6073-4d25-b438-e10c5fcca92c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:14:30,635 [275] [DEBUG] [app] Ending request: urn:request:62a73ab0-61aa-4096-a246-6a87fe6578f0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:62a73ab0-61aa-4096-a246-6a87fe6578f0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:14:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:14:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:30,638 [282] [DEBUG] [app] Starting request: urn:request:f316165a-a41d-4ac6-88c3-1eba5285a42c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:30,639 [284] [DEBUG] [app] Starting request: urn:request:25aea1e5-5e49-4626-963c-d3000564cf1d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:30,639 [282] [DEBUG] [app] Ending request: urn:request:f316165a-a41d-4ac6-88c3-1eba5285a42c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f316165a-a41d-4ac6-88c3-1eba5285a42c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:30,639 [284] [DEBUG] [app] Ending request: urn:request:25aea1e5-5e49-4626-963c-d3000564cf1d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:25aea1e5-5e49-4626-963c-d3000564cf1d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:14:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:30,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:30,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:30,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:30,659 [285] [DEBUG] [app] Ending request: urn:request:993413e9-3381-4287-a5b6-9523ebfe98d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:993413e9-3381-4287-a5b6-9523ebfe98d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:30,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:14:30,660 [286] [DEBUG] [app] Ending request: urn:request:f56b004e-7173-4261-81ed-bcabcfa69492 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f56b004e-7173-4261-81ed-bcabcfa69492', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:30,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:14:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:14:30,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:14:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:14:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:01.842666+00:00 (in 29.999447 seconds)
gcworker stdout | 2024-04-22 04:14:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:01 UTC)" (scheduled at 2024-04-22 04:14:31.842666+00:00)
gcworker stdout | 2024-04-22 04:14:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549671843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:14:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:14:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:14:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:14:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:14:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:29.931002+00:00 (in 57.996761 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:14:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:31 UTC)" (scheduled at 2024-04-22 04:14:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:14:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:14:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:14:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:42.430973+00:00 (in 9.999502 seconds)
notificationworker stdout | 2024-04-22 04:14:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:42 UTC)" (scheduled at 2024-04-22 04:14:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:14:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:14:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 32, 431875), True, datetime.datetime(2024, 4, 22, 4, 14, 32, 431875), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:14:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:14:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:14:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:14:35,719 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:14:37,028 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:14:39,548 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:14:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:14:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:10.238173+00:00 (in 29.999507 seconds)
autopruneworker stdout | 2024-04-22 04:14:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:10 UTC)" (scheduled at 2024-04-22 04:14:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:14:40,260 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759220259, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:14:40,265 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:14:40,265 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:14:40,265 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:14:40,518 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:14:40,864 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:14:41,103 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:14:41,574 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:14:42,229 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:14:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:52.430973+00:00 (in 9.999520 seconds)
notificationworker stdout | 2024-04-22 04:14:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:52 UTC)" (scheduled at 2024-04-22 04:14:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:14:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:14:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 42, 431744), True, datetime.datetime(2024, 4, 22, 4, 14, 42, 431744), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:14:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:14:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:14:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:14:52 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:14:42,530 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:14:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:14:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:57.806778+00:00 (in 14.999465 seconds)
quotatotalworker stdout | 2024-04-22 04:14:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:57 UTC)" (scheduled at 2024-04-22 04:14:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:14:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:14:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:14:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:14:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:14:43,770 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:14:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:14:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:47.065745+00:00 (in 2.998152 seconds)
securityworker stdout | 2024-04-22 04:14:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:14 UTC)" (scheduled at 2024-04-22 04:14:44.067050+00:00)
securityworker stdout | 2024-04-22 04:14:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:14:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:14:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:14:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:14:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:14:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:14:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:14:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:14:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 9, 44, 77873), 1, 2])
securityworker stdout | 2024-04-22 04:14:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 9, 44, 77873), 1, 2])
securityworker stdout | 2024-04-22 04:14:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:14:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:14:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:14:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:14:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:14:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:14:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:14:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:14:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:14:44,791 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:14:45,632 [285] [DEBUG] [app] Starting request: urn:request:7e83c674-16ac-4f53-8e1a-0fcc1486c7df (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:45,632 [284] [DEBUG] [app] Starting request: urn:request:c6baf132-8b6c-4a34-bb0c-c99fcc471fcd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:14:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:14:45,635 [275] [DEBUG] [app] Starting request: urn:request:ccc95711-2f73-4fe8-9ad7-ac630778da74 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:45,635 [283] [DEBUG] [app] Starting request: urn:request:9e2b38d8-4d6f-42f7-a9c2-1adc213f6e02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:14:45,635 [275] [DEBUG] [app] Ending request: urn:request:ccc95711-2f73-4fe8-9ad7-ac630778da74 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ccc95711-2f73-4fe8-9ad7-ac630778da74', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:14:45,635 [283] [DEBUG] [app] Ending request: urn:request:9e2b38d8-4d6f-42f7-a9c2-1adc213f6e02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e2b38d8-4d6f-42f7-a9c2-1adc213f6e02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:14:45,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:14:45,635 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:14:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:14:45,638 [282] [DEBUG] [app] Starting request: urn:request:a7f7db43-c958-43f5-a81f-e3839b95610e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:45,638 [285] [DEBUG] [app] Starting request: urn:request:a1595547-f766-40cc-b2c6-586db5492762 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:14:45,638 [282] [DEBUG] [app] Ending request: urn:request:a7f7db43-c958-43f5-a81f-e3839b95610e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a7f7db43-c958-43f5-a81f-e3839b95610e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:45,639 [285] [DEBUG] [app] Ending request: urn:request:a1595547-f766-40cc-b2c6-586db5492762 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1595547-f766-40cc-b2c6-586db5492762', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:14:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:14:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:14:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:14:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:14:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:14:45,639 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:45,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:14:45,647 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:14:45,659 [285] [DEBUG] [app] Ending request: urn:request:7e83c674-16ac-4f53-8e1a-0fcc1486c7df (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7e83c674-16ac-4f53-8e1a-0fcc1486c7df', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:45,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:14:45,659 [284] [DEBUG] [app] Ending request: urn:request:c6baf132-8b6c-4a34-bb0c-c99fcc471fcd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c6baf132-8b6c-4a34-bb0c-c99fcc471fcd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:14:45,659 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:14:45,659 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:14:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:14:45,659 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:14:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:14:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:14:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:14.067050+00:00 (in 27.000731 seconds)
securityworker stdout | 2024-04-22 04:14:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:17 UTC)" (scheduled at 2024-04-22 04:14:47.065745+00:00)
securityworker stdout | 2024-04-22 04:14:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:14:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:14:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:14:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:14:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:14:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:14:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:14:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:10.248668+00:00 (in 23.002223 seconds)
namespacegcworker stdout | 2024-04-22 04:14:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:47 UTC)" (scheduled at 2024-04-22 04:14:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:14:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:14:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 47, 246691), True, datetime.datetime(2024, 4, 22, 4, 14, 47, 246691), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:14:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:14:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:14:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:14:47,290 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:14:47,437 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:14:47,961 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:14:48,543 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:14:49,536 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:14:49,991 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:14:49,995 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:14:49,998 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:14:50,661 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:14:51,022 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:14:51,531 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:14:51,592 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:14:51,595 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:14:51,598 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:14:51,602 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:14:51,607 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:14:51,610 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:14:51,615 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:14:51,623 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:14:51,637 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:14:51,647 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:14:51,650 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:14:51,658 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:14:51,660 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:14:52,340 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:14:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:14:54.433683+00:00 (in 2.002218 seconds)
notificationworker stdout | 2024-04-22 04:14:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:02 UTC)" (scheduled at 2024-04-22 04:14:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:14:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:14:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 52, 431895), True, datetime.datetime(2024, 4, 22, 4, 14, 52, 431895), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:14:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:14:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:14:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:02 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:14:52,556 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:14:52,620 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:14:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:14:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:18.047111+00:00 (in 25.001045 seconds)
exportactionlogsworker stdout | 2024-04-22 04:14:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:53 UTC)" (scheduled at 2024-04-22 04:14:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:14:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:14:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 14, 53, 46343), True, datetime.datetime(2024, 4, 22, 4, 14, 53, 46343), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:14:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:14:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:14:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:14:53,487 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:14:53,489 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:14:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:14:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:02.430973+00:00 (in 7.996806 seconds)
notificationworker stdout | 2024-04-22 04:14:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:54 UTC)" (scheduled at 2024-04-22 04:14:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:14:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:14:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:15:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:14:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:14:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:12.806778+00:00 (in 14.999482 seconds)
quotatotalworker stdout | 2024-04-22 04:14:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:12 UTC)" (scheduled at 2024-04-22 04:14:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:14:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:14:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:14:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:14:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:14:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:28.130973+00:00 (in 29.999460 seconds)
buildlogsarchiver stdout | 2024-04-22 04:14:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:28 UTC)" (scheduled at 2024-04-22 04:14:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:14:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 14, 58, 131919), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:14:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:14:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:14:58,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:15:00,632 [282] [DEBUG] [app] Starting request: urn:request:466f7c1d-725e-4b32-bb5b-097c1f61cbed (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:00,632 [286] [DEBUG] [app] Starting request: urn:request:28837e4b-7ac8-4b20-b144-8fe2db2d9357 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:15:00,635 [283] [DEBUG] [app] Starting request: urn:request:2a4ec5f1-d131-45af-aaf2-dd570d201e23 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:00,635 [283] [DEBUG] [app] Ending request: urn:request:2a4ec5f1-d131-45af-aaf2-dd570d201e23 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2a4ec5f1-d131-45af-aaf2-dd570d201e23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:00,636 [275] [DEBUG] [app] Starting request: urn:request:9d872ad1-b103-41a1-a764-ffcca5c912ae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:15:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 04:15:00,636 [275] [DEBUG] [app] Ending request: urn:request:9d872ad1-b103-41a1-a764-ffcca5c912ae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9d872ad1-b103-41a1-a764-ffcca5c912ae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:15:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:15:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:00,639 [282] [DEBUG] [app] Starting request: urn:request:4d8b06a5-29db-41b6-a315-0544728315d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:15:00,639 [282] [DEBUG] [app] Ending request: urn:request:4d8b06a5-29db-41b6-a315-0544728315d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:4d8b06a5-29db-41b6-a315-0544728315d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:15:00,640 [285] [DEBUG] [app] Starting request: urn:request:174f3729-c02b-428d-bc0b-2565bb70f233 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:00,640 [285] [DEBUG] [app] Ending request: urn:request:174f3729-c02b-428d-bc0b-2565bb70f233 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:174f3729-c02b-428d-bc0b-2565bb70f233', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:00,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:00,661 [282] [DEBUG] [app] Ending request: urn:request:466f7c1d-725e-4b32-bb5b-097c1f61cbed (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:466f7c1d-725e-4b32-bb5b-097c1f61cbed', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:15:00,671 [286] [DEBUG] [app] Ending request: urn:request:28837e4b-7ac8-4b20-b144-8fe2db2d9357 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:28837e4b-7ac8-4b20-b144-8fe2db2d9357', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:00,672 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.041 118 0.041)
gunicorn-web stdout | 2024-04-22 04:15:00,672 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:15:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:15:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:31.842666+00:00 (in 29.999492 seconds)
gcworker stdout | 2024-04-22 04:15:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:31 UTC)" (scheduled at 2024-04-22 04:15:01.842666+00:00)
gcworker stdout | 2024-04-22 04:15:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549701843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:15:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:15:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:15:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:15:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:15:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:09.033206+00:00 (in 6.997076 seconds)
repositorygcworker stdout | 2024-04-22 04:15:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:02 UTC)" (scheduled at 2024-04-22 04:15:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:15:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:15:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:15:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:12.430973+00:00 (in 9.999552 seconds)
notificationworker stdout | 2024-04-22 04:15:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:12 UTC)" (scheduled at 2024-04-22 04:15:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:15:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:15:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 2, 431831), True, datetime.datetime(2024, 4, 22, 4, 15, 2, 431831), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:15:02,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:15:02,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:15:02,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:15:05,741 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:15:07,047 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:15:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:15:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:02.035613+00:00 (in 53.001913 seconds)
repositorygcworker stdout | 2024-04-22 04:15:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:09 UTC)" (scheduled at 2024-04-22 04:15:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:15:09,033 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:15:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 9, 33994), True, datetime.datetime(2024, 4, 22, 4, 15, 9, 33994), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:15:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:15:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:15:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:15:09,573 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:15:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:15:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:40.238173+00:00 (in 29.999490 seconds)
autopruneworker stdout | 2024-04-22 04:15:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:40 UTC)" (scheduled at 2024-04-22 04:15:10.238173+00:00)
namespacegcworker stdout | 2024-04-22 04:15:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:15:10,249 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759250248, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:15:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:47.245913+00:00 (in 36.996741 seconds)
namespacegcworker stdout | 2024-04-22 04:15:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:10 UTC)" (scheduled at 2024-04-22 04:15:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:15:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:15:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:15:10,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:15:10,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:15:10,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:15:10,547 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:15:10,900 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:15:11,119 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:15:11,583 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:15:12,266 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:15:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:22.430973+00:00 (in 9.999587 seconds)
notificationworker stdout | 2024-04-22 04:15:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:22 UTC)" (scheduled at 2024-04-22 04:15:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:15:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:15:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 12, 431756), True, datetime.datetime(2024, 4, 22, 4, 15, 12, 431756), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:15:12,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:15:12,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:15:12,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:22 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:15:12,548 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:15:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:15:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:27.806778+00:00 (in 14.999485 seconds)
quotatotalworker stdout | 2024-04-22 04:15:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:27 UTC)" (scheduled at 2024-04-22 04:15:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:15:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:15:12,820 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:15:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:15:13,799 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:15:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:15:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:17.065745+00:00 (in 2.998112 seconds)
securityworker stdout | 2024-04-22 04:15:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:44 UTC)" (scheduled at 2024-04-22 04:15:14.067050+00:00)
securityworker stdout | 2024-04-22 04:15:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:15:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:15:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:15:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:15:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:15:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:15:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:15:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:15:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 10, 14, 78226), 1, 2])
securityworker stdout | 2024-04-22 04:15:14,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 10, 14, 78226), 1, 2])
securityworker stdout | 2024-04-22 04:15:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:15:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:15:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:15:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:15:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:15:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:15:14,815 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:15:15,632 [282] [DEBUG] [app] Starting request: urn:request:81ac872d-6051-4dfc-a2e1-2f6e2593fe01 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:15,632 [286] [DEBUG] [app] Starting request: urn:request:fc68fdb8-2e8c-49fe-b67d-3df9ac140484 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:15:15,635 [275] [DEBUG] [app] Starting request: urn:request:6a636f61-29eb-4e10-a86b-610488d3b3ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:15,635 [277] [DEBUG] [app] Starting request: urn:request:fc1881c8-386d-4d46-a4b3-69c2aa707e28 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:15,636 [275] [DEBUG] [app] Ending request: urn:request:6a636f61-29eb-4e10-a86b-610488d3b3ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6a636f61-29eb-4e10-a86b-610488d3b3ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:15,636 [277] [DEBUG] [app] Ending request: urn:request:fc1881c8-386d-4d46-a4b3-69c2aa707e28 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fc1881c8-386d-4d46-a4b3-69c2aa707e28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:15:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:15:15,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:15:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:15,639 [282] [DEBUG] [app] Starting request: urn:request:d6cb5b31-ba88-4160-a0e7-067fb978ca1a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:15:15,639 [282] [DEBUG] [app] Ending request: urn:request:d6cb5b31-ba88-4160-a0e7-067fb978ca1a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d6cb5b31-ba88-4160-a0e7-067fb978ca1a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:15,639 [284] [DEBUG] [app] Starting request: urn:request:dc9c6d58-e201-4c34-9d1c-9267ec8629ab (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:15:15,639 [284] [DEBUG] [app] Ending request: urn:request:dc9c6d58-e201-4c34-9d1c-9267ec8629ab (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:dc9c6d58-e201-4c34-9d1c-9267ec8629ab', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:15,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:15,659 [282] [DEBUG] [app] Ending request: urn:request:81ac872d-6051-4dfc-a2e1-2f6e2593fe01 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81ac872d-6051-4dfc-a2e1-2f6e2593fe01', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:15,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:15,659 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:15:15,659 [286] [DEBUG] [app] Ending request: urn:request:fc68fdb8-2e8c-49fe-b67d-3df9ac140484 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fc68fdb8-2e8c-49fe-b67d-3df9ac140484', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:15:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:15:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:44.067050+00:00 (in 27.000752 seconds)
securityworker stdout | 2024-04-22 04:15:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:47 UTC)" (scheduled at 2024-04-22 04:15:17.065745+00:00)
securityworker stdout | 2024-04-22 04:15:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:15:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:15:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:15:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:15:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:15:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:15:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 10, 17, 70398), 1, 2])
securityworker stdout | 2024-04-22 04:15:17,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:17,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 10, 17, 70398), 1, 2])
securityworker stdout | 2024-04-22 04:15:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:17,099 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:15:17,099 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:15:17,326 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:15:17,450 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:15:17,997 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:15:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:15:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:53.045554+00:00 (in 34.997941 seconds)
exportactionlogsworker stdout | 2024-04-22 04:15:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:18 UTC)" (scheduled at 2024-04-22 04:15:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:15:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:15:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:18 UTC)" executed successfully
exportactionlogsworker stdout | 2024-04-22 04:15:18,556 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:15:19,566 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:15:19,999 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:15:20,002 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:15:20,005 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:15:20,698 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:15:21,060 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:15:21,542 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:15:21,603 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:15:21,609 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:15:21,612 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:15:21,616 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:15:21,619 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:15:21,621 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:15:21,624 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:15:21,633 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:15:21,643 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:15:21,654 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:15:21,657 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:15:21,665 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:15:21,667 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:15:22,365 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:15:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:32.430973+00:00 (in 9.999528 seconds)
notificationworker stdout | 2024-04-22 04:15:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:32 UTC)" (scheduled at 2024-04-22 04:15:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:15:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:15:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 22, 431836), True, datetime.datetime(2024, 4, 22, 4, 15, 22, 431836), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:15:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:15:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:15:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:32 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:15:22,592 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:15:22,634 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:15:23,524 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:15:23,526 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:15:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:15:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:24.738365+00:00 (in 59.999493 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:15:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:24 UTC)" (scheduled at 2024-04-22 04:15:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:15:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:15:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:15:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:15:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:15:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:42.806778+00:00 (in 14.999494 seconds)
quotatotalworker stdout | 2024-04-22 04:15:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:42 UTC)" (scheduled at 2024-04-22 04:15:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:15:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:15:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:15:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:15:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:15:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:58.130973+00:00 (in 29.999469 seconds)
buildlogsarchiver stdout | 2024-04-22 04:15:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:58 UTC)" (scheduled at 2024-04-22 04:15:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:15:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 15, 28, 131897), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:15:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:15:28,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:15:28,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:15:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:31.933721+00:00 (in 2.002198 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:29 UTC)" (scheduled at 2024-04-22 04:15:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 29, 931923), True, datetime.datetime(2024, 4, 22, 4, 15, 29, 931923), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:15:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:15:30,632 [286] [DEBUG] [app] Starting request: urn:request:9e52e4c4-41f8-422f-a568-2d9df3dd9721 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:30,632 [282] [DEBUG] [app] Starting request: urn:request:a6038327-af67-4678-ad86-9d9908416647 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:15:30,635 [281] [DEBUG] [app] Starting request: urn:request:153815f3-7e94-48c3-8db5-8d3f17fd51b5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:30,635 [275] [DEBUG] [app] Starting request: urn:request:a77070c6-048a-49b0-aed5-89d988a0e7f6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:30,635 [281] [DEBUG] [app] Ending request: urn:request:153815f3-7e94-48c3-8db5-8d3f17fd51b5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:153815f3-7e94-48c3-8db5-8d3f17fd51b5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:30,636 [275] [DEBUG] [app] Ending request: urn:request:a77070c6-048a-49b0-aed5-89d988a0e7f6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a77070c6-048a-49b0-aed5-89d988a0e7f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:15:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:15:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:30,638 [282] [DEBUG] [app] Starting request: urn:request:6625246c-daa8-483b-b9af-0a09496b92c6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:15:30,639 [282] [DEBUG] [app] Ending request: urn:request:6625246c-daa8-483b-b9af-0a09496b92c6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6625246c-daa8-483b-b9af-0a09496b92c6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:30,639 [285] [DEBUG] [app] Starting request: urn:request:8a44e053-0e48-4e2d-abf8-33cca68c4d6d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:30,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:15:30,639 [285] [DEBUG] [app] Ending request: urn:request:8a44e053-0e48-4e2d-abf8-33cca68c4d6d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8a44e053-0e48-4e2d-abf8-33cca68c4d6d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:30,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:30,659 [282] [DEBUG] [app] Ending request: urn:request:a6038327-af67-4678-ad86-9d9908416647 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a6038327-af67-4678-ad86-9d9908416647', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:30,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:30,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:15:30,660 [286] [DEBUG] [app] Ending request: urn:request:9e52e4c4-41f8-422f-a568-2d9df3dd9721 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:9e52e4c4-41f8-422f-a568-2d9df3dd9721', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:30,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:15:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:15:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:01.842666+00:00 (in 29.999515 seconds)
gcworker stdout | 2024-04-22 04:15:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:01 UTC)" (scheduled at 2024-04-22 04:15:31.842666+00:00)
gcworker stdout | 2024-04-22 04:15:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549731843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:15:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:15:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:15:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:15:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:15:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:29.931002+00:00 (in 57.996779 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:15:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:31 UTC)" (scheduled at 2024-04-22 04:15:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:15:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:15:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:15:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:42.430973+00:00 (in 9.999517 seconds)
notificationworker stdout | 2024-04-22 04:15:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:42 UTC)" (scheduled at 2024-04-22 04:15:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:15:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:15:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 32, 431911), True, datetime.datetime(2024, 4, 22, 4, 15, 32, 431911), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:15:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:15:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:15:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:15:35,755 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:15:37,083 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:15:39,591 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:15:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:15:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:10.238173+00:00 (in 29.999437 seconds)
autopruneworker stdout | 2024-04-22 04:15:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:10 UTC)" (scheduled at 2024-04-22 04:15:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:15:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759280247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:15:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:15:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:15:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:15:40,585 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:15:40,916 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:15:41,143 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:15:41,619 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:15:42,298 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:15:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:52.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 04:15:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:52 UTC)" (scheduled at 2024-04-22 04:15:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:15:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:15:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 42, 431915), True, datetime.datetime(2024, 4, 22, 4, 15, 42, 431915), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:15:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:15:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:15:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:15:52 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:15:42,585 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:15:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:15:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:57.806778+00:00 (in 14.999438 seconds)
quotatotalworker stdout | 2024-04-22 04:15:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:57 UTC)" (scheduled at 2024-04-22 04:15:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:15:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:15:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:15:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:15:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:15:43,816 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:15:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:15:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:47.065745+00:00 (in 2.998143 seconds)
securityworker stdout | 2024-04-22 04:15:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:14 UTC)" (scheduled at 2024-04-22 04:15:44.067050+00:00)
securityworker stdout | 2024-04-22 04:15:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:15:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:15:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:15:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:15:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:15:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:15:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:15:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:15:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:44,107 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 10, 44, 77664), 1, 2])
securityworker stdout | 2024-04-22 04:15:44,111 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:44,111 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:44,112 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 10, 44, 77664), 1, 2])
securityworker stdout | 2024-04-22 04:15:44,115 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:15:44,115 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:15:44,115 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:44,116 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:15:44,116 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:15:44,116 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:15:44,116 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:15:44,117 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:15:44,117 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:15:44,117 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:15:44,828 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:15:45,632 [284] [DEBUG] [app] Starting request: urn:request:6ab7d995-4f9c-46ca-a8ba-0dcf0b3ad910 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:45,632 [286] [DEBUG] [app] Starting request: urn:request:115c3aec-b1a8-4083-87fd-b95bd27f0f25 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:15:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:15:45,635 [280] [DEBUG] [app] Starting request: urn:request:cbd1304d-1328-42ec-87cf-b94b023a6ed5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:45,635 [281] [DEBUG] [app] Starting request: urn:request:b07149f7-3472-4196-90ff-85b175f21e60 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:15:45,635 [280] [DEBUG] [app] Ending request: urn:request:cbd1304d-1328-42ec-87cf-b94b023a6ed5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cbd1304d-1328-42ec-87cf-b94b023a6ed5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:45,635 [281] [DEBUG] [app] Ending request: urn:request:b07149f7-3472-4196-90ff-85b175f21e60 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b07149f7-3472-4196-90ff-85b175f21e60', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:15:45,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:15:45,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:15:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:15:45,638 [282] [DEBUG] [app] Starting request: urn:request:1086adaf-651e-4232-b9ef-2f98637e52ad (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:15:45,638 [285] [DEBUG] [app] Starting request: urn:request:70be4f78-37b6-448a-b6fa-7635f221a6d6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:15:45,639 [282] [DEBUG] [app] Ending request: urn:request:1086adaf-651e-4232-b9ef-2f98637e52ad (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1086adaf-651e-4232-b9ef-2f98637e52ad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:45,639 [285] [DEBUG] [app] Ending request: urn:request:70be4f78-37b6-448a-b6fa-7635f221a6d6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:70be4f78-37b6-448a-b6fa-7635f221a6d6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:15:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:15:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:15:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:15:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:15:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:15:45,639 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:15:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:15:45,659 [284] [DEBUG] [app] Ending request: urn:request:6ab7d995-4f9c-46ca-a8ba-0dcf0b3ad910 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6ab7d995-4f9c-46ca-a8ba-0dcf0b3ad910', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:15:45,659 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:45,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:15:45,660 [286] [DEBUG] [app] Ending request: urn:request:115c3aec-b1a8-4083-87fd-b95bd27f0f25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:115c3aec-b1a8-4083-87fd-b95bd27f0f25', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:15:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:15:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:15:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:15:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:15:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:15:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:14.067050+00:00 (in 27.000765 seconds)
securityworker stdout | 2024-04-22 04:15:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:17 UTC)" (scheduled at 2024-04-22 04:15:47.065745+00:00)
securityworker stdout | 2024-04-22 04:15:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:15:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:15:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:15:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:15:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:15:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:15:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:15:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:10.248668+00:00 (in 23.002243 seconds)
namespacegcworker stdout | 2024-04-22 04:15:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:47 UTC)" (scheduled at 2024-04-22 04:15:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:15:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:15:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 47, 246686), True, datetime.datetime(2024, 4, 22, 4, 15, 47, 246686), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:15:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:15:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:15:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:15:47,341 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:15:47,487 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:15:48,034 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:15:48,585 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:15:49,592 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:15:50,007 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:15:50,010 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:15:50,013 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:15:50,735 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:15:51,097 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:15:51,552 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:15:51,615 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:15:51,620 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:15:51,623 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:15:51,627 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:15:51,630 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:15:51,633 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:15:51,635 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:15:51,642 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:15:51,650 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:15:51,662 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:15:51,665 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:15:51,671 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:15:51,674 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
blobuploadcleanupworker stdout | 2024-04-22 04:15:52,402 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:15:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:15:54.433683+00:00 (in 2.002179 seconds)
notificationworker stdout | 2024-04-22 04:15:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:02 UTC)" (scheduled at 2024-04-22 04:15:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:15:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:15:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 52, 431945), True, datetime.datetime(2024, 4, 22, 4, 15, 52, 431945), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:15:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:15:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:15:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:02 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:15:52,628 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:15:52,671 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:15:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:15:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:18.047111+00:00 (in 25.001083 seconds)
exportactionlogsworker stdout | 2024-04-22 04:15:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:53 UTC)" (scheduled at 2024-04-22 04:15:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:15:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:15:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 15, 53, 46497), True, datetime.datetime(2024, 4, 22, 4, 15, 53, 46497), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:15:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:15:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:15:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:53 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:15:53,560 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:15:53,563 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:15:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:15:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:02.430973+00:00 (in 7.996767 seconds)
notificationworker stdout | 2024-04-22 04:15:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:54 UTC)" (scheduled at 2024-04-22 04:15:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:15:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:15:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:16:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:15:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:15:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:12.806778+00:00 (in 14.999495 seconds)
quotatotalworker stdout | 2024-04-22 04:15:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:12 UTC)" (scheduled at 2024-04-22 04:15:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:15:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:15:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:15:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:15:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:15:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:28.130973+00:00 (in 29.999517 seconds)
buildlogsarchiver stdout | 2024-04-22 04:15:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:28 UTC)" (scheduled at 2024-04-22 04:15:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:15:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 15, 58, 131808), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:15:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:15:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:15:58,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:16:00,632 [284] [DEBUG] [app] Starting request: urn:request:601d8da2-3b55-49ac-a011-628c9eff1b19 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:00,632 [282] [DEBUG] [app] Starting request: urn:request:b5ac67bb-add4-4379-b242-4b4d8414e8b0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:00,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:16:00,635 [280] [DEBUG] [app] Starting request: urn:request:a5af28bd-5fa7-45b4-b095-4ea119df2c29 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:00,635 [275] [DEBUG] [app] Starting request: urn:request:cdce6c60-2a6d-4e28-84be-a2c22f341621 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:00,636 [280] [DEBUG] [app] Ending request: urn:request:a5af28bd-5fa7-45b4-b095-4ea119df2c29 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a5af28bd-5fa7-45b4-b095-4ea119df2c29', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:00,636 [275] [DEBUG] [app] Ending request: urn:request:cdce6c60-2a6d-4e28-84be-a2c22f341621 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cdce6c60-2a6d-4e28-84be-a2c22f341621', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:00,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:16:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:00,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 04:16:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:00,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:00,638 [282] [DEBUG] [app] Starting request: urn:request:93083ac8-a059-4a13-9d35-752fbad45e65 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:00,639 [282] [DEBUG] [app] Ending request: urn:request:93083ac8-a059-4a13-9d35-752fbad45e65 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:93083ac8-a059-4a13-9d35-752fbad45e65', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:16:00,639 [286] [DEBUG] [app] Starting request: urn:request:d0d445f0-2219-48af-bf68-d268adb3e610 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:16:00,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:00,639 [286] [DEBUG] [app] Ending request: urn:request:d0d445f0-2219-48af-bf68-d268adb3e610 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d0d445f0-2219-48af-bf68-d268adb3e610', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:16:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:00,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:16:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:00,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:00,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:00,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:00,660 [282] [DEBUG] [app] Ending request: urn:request:b5ac67bb-add4-4379-b242-4b4d8414e8b0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b5ac67bb-add4-4379-b242-4b4d8414e8b0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:00,660 [284] [DEBUG] [app] Ending request: urn:request:601d8da2-3b55-49ac-a011-628c9eff1b19 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:601d8da2-3b55-49ac-a011-628c9eff1b19', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:00,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:00,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:16:00,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:16:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:16:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:31.842666+00:00 (in 29.999500 seconds)
gcworker stdout | 2024-04-22 04:16:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:31 UTC)" (scheduled at 2024-04-22 04:16:01.842666+00:00)
gcworker stdout | 2024-04-22 04:16:01,843 [67] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2024-04-22 04:16:01,857 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549761856, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:16:01,862 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:16:01,862 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:16:01,862 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:16:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:16:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:09.033206+00:00 (in 6.997042 seconds)
repositorygcworker stdout | 2024-04-22 04:16:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:02 UTC)" (scheduled at 2024-04-22 04:16:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:16:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:16:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:16:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:12.430973+00:00 (in 9.999433 seconds)
notificationworker stdout | 2024-04-22 04:16:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:12 UTC)" (scheduled at 2024-04-22 04:16:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:16:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:16:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 2, 432003), True, datetime.datetime(2024, 4, 22, 4, 16, 2, 432003), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:16:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:16:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:16:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:16:05,775 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:16:07,120 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:16:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:16:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:02.035613+00:00 (in 53.001940 seconds)
repositorygcworker stdout | 2024-04-22 04:16:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:09 UTC)" (scheduled at 2024-04-22 04:16:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:16:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:16:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 9, 34098), True, datetime.datetime(2024, 4, 22, 4, 16, 9, 34098), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:16:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:16:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:16:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:16:09,625 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:16:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:16:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:40.238173+00:00 (in 29.999414 seconds)
autopruneworker stdout | 2024-04-22 04:16:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:40 UTC)" (scheduled at 2024-04-22 04:16:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:16:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759310247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:16:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:16:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:30.247852+00:00 (in 19.998719 seconds)
namespacegcworker stdout | 2024-04-22 04:16:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:10 UTC)" (scheduled at 2024-04-22 04:16:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:16:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:16:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:16:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:16:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:16:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:16:10,621 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:16:10,939 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:16:11,179 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:16:11,656 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:16:12,334 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:16:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:22.430973+00:00 (in 9.999420 seconds)
notificationworker stdout | 2024-04-22 04:16:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:22 UTC)" (scheduled at 2024-04-22 04:16:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:16:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:16:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 12, 432014), True, datetime.datetime(2024, 4, 22, 4, 16, 12, 432014), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:16:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:16:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:16:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:22 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:16:12,622 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:16:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:16:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:27.806778+00:00 (in 14.999450 seconds)
quotatotalworker stdout | 2024-04-22 04:16:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:27 UTC)" (scheduled at 2024-04-22 04:16:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:16:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:16:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:16:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:16:13,848 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:16:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:16:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:17.065745+00:00 (in 2.998129 seconds)
securityworker stdout | 2024-04-22 04:16:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:44 UTC)" (scheduled at 2024-04-22 04:16:14.067050+00:00)
securityworker stdout | 2024-04-22 04:16:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:16:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:16:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:16:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:16:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:16:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:16:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:16:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:16:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 11, 14, 77908), 1, 2])
securityworker stdout | 2024-04-22 04:16:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 11, 14, 77908), 1, 2])
securityworker stdout | 2024-04-22 04:16:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:16:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:16:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:16:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:16:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:16:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:16:14,864 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:16:15,632 [282] [DEBUG] [app] Starting request: urn:request:4cde5876-aca3-4bd8-b84c-3fa50e98dbfe (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:15,632 [284] [DEBUG] [app] Starting request: urn:request:148775fe-a648-4297-818d-188577c2b986 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:16:15,635 [278] [DEBUG] [app] Starting request: urn:request:5d5c291f-8026-4965-a86e-ab098ba82637 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:15,636 [280] [DEBUG] [app] Starting request: urn:request:1d7c950c-cf72-4855-9a6a-ad66279c43f4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:15,636 [278] [DEBUG] [app] Ending request: urn:request:5d5c291f-8026-4965-a86e-ab098ba82637 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5d5c291f-8026-4965-a86e-ab098ba82637', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:15,636 [280] [DEBUG] [app] Ending request: urn:request:1d7c950c-cf72-4855-9a6a-ad66279c43f4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:1d7c950c-cf72-4855-9a6a-ad66279c43f4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:15,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:16:15,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:15,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:15,639 [282] [DEBUG] [app] Starting request: urn:request:b0aaa970-436c-49cc-aa26-63e83b02a7f6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:15,639 [282] [DEBUG] [app] Ending request: urn:request:b0aaa970-436c-49cc-aa26-63e83b02a7f6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b0aaa970-436c-49cc-aa26-63e83b02a7f6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:16:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:16:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:15,640 [286] [DEBUG] [app] Starting request: urn:request:1f474867-fff3-40b6-942b-abd0eb935039 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:15,640 [286] [DEBUG] [app] Ending request: urn:request:1f474867-fff3-40b6-942b-abd0eb935039 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1f474867-fff3-40b6-942b-abd0eb935039', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:16:15,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:15,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:16:15,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:15,659 [282] [DEBUG] [app] Ending request: urn:request:4cde5876-aca3-4bd8-b84c-3fa50e98dbfe (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4cde5876-aca3-4bd8-b84c-3fa50e98dbfe', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:15,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:16:15,660 [284] [DEBUG] [app] Ending request: urn:request:148775fe-a648-4297-818d-188577c2b986 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:148775fe-a648-4297-818d-188577c2b986', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:15,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:16:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:16:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:44.067050+00:00 (in 27.000774 seconds)
securityworker stdout | 2024-04-22 04:16:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:47 UTC)" (scheduled at 2024-04-22 04:16:17.065745+00:00)
securityworker stdout | 2024-04-22 04:16:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:16:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:16:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:16:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:16:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:16:17,087 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:16:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:17,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 11, 17, 70932), 1, 2])
securityworker stdout | 2024-04-22 04:16:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 11, 17, 70932), 1, 2])
securityworker stdout | 2024-04-22 04:16:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:16:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:16:17,375 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:16:17,523 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:16:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:16:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:53.045554+00:00 (in 34.997929 seconds)
exportactionlogsworker stdout | 2024-04-22 04:16:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:18 UTC)" (scheduled at 2024-04-22 04:16:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:16:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:16:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:16:18,070 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:16:18,623 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:16:19,620 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:16:20,015 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:16:20,017 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:16:20,021 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:16:20,768 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:16:21,133 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:16:21,561 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:16:21,624 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:16:21,630 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:16:21,633 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:16:21,636 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:16:21,638 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:16:21,642 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:16:21,644 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:16:21,649 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:16:21,657 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:16:21,670 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:16:21,672 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:16:21,679 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:16:21,682 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:16:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:32.430973+00:00 (in 9.999500 seconds)
notificationworker stdout | 2024-04-22 04:16:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:32 UTC)" (scheduled at 2024-04-22 04:16:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:16:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:16:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 22, 431873), True, datetime.datetime(2024, 4, 22, 4, 16, 22, 431873), 0, 'notification/%', 50, 1, 0])
blobuploadcleanupworker stdout | 2024-04-22 04:16:22,437 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
notificationworker stdout | 2024-04-22 04:16:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:16:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:16:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:32 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:16:22,666 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:16:22,707 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:16:23,577 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:16:23,597 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:16:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:16:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:24.738365+00:00 (in 59.999506 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:16:24,738 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:24 UTC)" (scheduled at 2024-04-22 04:16:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:16:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:16:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:16:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:16:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:16:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:42.806778+00:00 (in 14.999448 seconds)
quotatotalworker stdout | 2024-04-22 04:16:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:42 UTC)" (scheduled at 2024-04-22 04:16:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:16:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:16:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:16:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:16:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:16:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:58.130973+00:00 (in 29.999476 seconds)
buildlogsarchiver stdout | 2024-04-22 04:16:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:58 UTC)" (scheduled at 2024-04-22 04:16:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:16:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 16, 28, 131895), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:16:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:16:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:16:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:16:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:31.933721+00:00 (in 2.002189 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:29 UTC)" (scheduled at 2024-04-22 04:16:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 29, 931934), True, datetime.datetime(2024, 4, 22, 4, 16, 29, 931934), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:16:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:16:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:16:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:47.245913+00:00 (in 16.997545 seconds)
namespacegcworker stdout | 2024-04-22 04:16:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:21:30 UTC)" (scheduled at 2024-04-22 04:16:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 04:16:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 16, 30, 248745), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:16:30,262 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 16, 30, 248745), True, datetime.datetime(2024, 4, 22, 4, 16, 30, 248745), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:16:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 16, 30, 248745), True, datetime.datetime(2024, 4, 22, 4, 16, 30, 248745), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 4, 16, 30, 248745), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:16:30,270 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:16:30,270 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:21:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:16:30,632 [285] [DEBUG] [app] Starting request: urn:request:a2d7d231-7bcd-496c-97d5-d273f0cb38e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:30,632 [282] [DEBUG] [app] Starting request: urn:request:e01b2a3b-2005-45ff-98ed-8f55ec7686fd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:16:30,635 [283] [DEBUG] [app] Starting request: urn:request:08bd71c5-c951-4dc8-9a6c-9d7f76951c6a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:30,635 [276] [DEBUG] [app] Starting request: urn:request:0c05d5ad-b5a4-431d-b8c1-fdf4c7e2356e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:30,636 [276] [DEBUG] [app] Ending request: urn:request:0c05d5ad-b5a4-431d-b8c1-fdf4c7e2356e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c05d5ad-b5a4-431d-b8c1-fdf4c7e2356e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:30,636 [283] [DEBUG] [app] Ending request: urn:request:08bd71c5-c951-4dc8-9a6c-9d7f76951c6a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:08bd71c5-c951-4dc8-9a6c-9d7f76951c6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:30,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:16:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 04:16:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:30,639 [286] [DEBUG] [app] Starting request: urn:request:651dc94a-7b9e-4f04-bfab-0071d643f610 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:30,639 [284] [DEBUG] [app] Starting request: urn:request:e537c95b-2c25-44ef-aa9e-9eefc5a1f513 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:30,639 [286] [DEBUG] [app] Ending request: urn:request:651dc94a-7b9e-4f04-bfab-0071d643f610 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:651dc94a-7b9e-4f04-bfab-0071d643f610', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:16:30,639 [284] [DEBUG] [app] Ending request: urn:request:e537c95b-2c25-44ef-aa9e-9eefc5a1f513 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e537c95b-2c25-44ef-aa9e-9eefc5a1f513', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:16:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:30,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:30,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:30,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:30,660 [285] [DEBUG] [app] Ending request: urn:request:a2d7d231-7bcd-496c-97d5-d273f0cb38e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a2d7d231-7bcd-496c-97d5-d273f0cb38e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:30,660 [282] [DEBUG] [app] Ending request: urn:request:e01b2a3b-2005-45ff-98ed-8f55ec7686fd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e01b2a3b-2005-45ff-98ed-8f55ec7686fd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:30,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:30,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:16:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:16:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:16:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:01.842666+00:00 (in 29.999497 seconds)
gcworker stdout | 2024-04-22 04:16:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:01 UTC)" (scheduled at 2024-04-22 04:16:31.842666+00:00)
gcworker stdout | 2024-04-22 04:16:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549791843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:16:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:16:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:16:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:16:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:16:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:29.931002+00:00 (in 57.996765 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:16:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:31 UTC)" (scheduled at 2024-04-22 04:16:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:16:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:16:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:16:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:42.430973+00:00 (in 9.999542 seconds)
notificationworker stdout | 2024-04-22 04:16:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:42 UTC)" (scheduled at 2024-04-22 04:16:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:16:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:16:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 32, 431692), True, datetime.datetime(2024, 4, 22, 4, 16, 32, 431692), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:16:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:16:32,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:16:32,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:16:35,811 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:16:37,156 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:16:39,637 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:16:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:16:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:10.238173+00:00 (in 29.999510 seconds)
autopruneworker stdout | 2024-04-22 04:16:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:10 UTC)" (scheduled at 2024-04-22 04:16:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:16:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759340247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:16:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:16:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:16:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:16:40,653 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:16:40,975 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:16:41,215 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:16:41,675 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:16:42,346 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:16:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:52.430973+00:00 (in 9.999528 seconds)
notificationworker stdout | 2024-04-22 04:16:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:52 UTC)" (scheduled at 2024-04-22 04:16:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:16:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:16:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 42, 431696), True, datetime.datetime(2024, 4, 22, 4, 16, 42, 431696), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:16:42,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:16:42,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:16:42,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:16:52 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:16:42,637 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:16:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:16:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:57.806778+00:00 (in 14.999452 seconds)
quotatotalworker stdout | 2024-04-22 04:16:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:57 UTC)" (scheduled at 2024-04-22 04:16:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:16:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:16:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:16:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:16:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:16:43,885 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:16:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:16:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:47.065745+00:00 (in 2.998154 seconds)
securityworker stdout | 2024-04-22 04:16:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:14 UTC)" (scheduled at 2024-04-22 04:16:44.067050+00:00)
securityworker stdout | 2024-04-22 04:16:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:16:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:16:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:16:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:16:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:16:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:16:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:16:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:16:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 11, 44, 77996), 1, 2])
securityworker stdout | 2024-04-22 04:16:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 11, 44, 77996), 1, 2])
securityworker stdout | 2024-04-22 04:16:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:16:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:16:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:16:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:16:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:16:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:16:44,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:16:44,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:16:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:16:44,885 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:16:45,632 [284] [DEBUG] [app] Starting request: urn:request:72db79d7-4ac0-48d8-8dd3-51419b7b1823 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:45,632 [286] [DEBUG] [app] Starting request: urn:request:d1870465-5a49-47af-b49c-698a9530a58c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:16:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:16:45,635 [275] [DEBUG] [app] Starting request: urn:request:131e11fa-1f8c-4514-b618-672dc2b6ca80 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:45,635 [280] [DEBUG] [app] Starting request: urn:request:3b1f65f4-ee1d-4ebe-8c79-d5fdfb56aa02 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:16:45,635 [275] [DEBUG] [app] Ending request: urn:request:131e11fa-1f8c-4514-b618-672dc2b6ca80 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:131e11fa-1f8c-4514-b618-672dc2b6ca80', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:45,636 [280] [DEBUG] [app] Ending request: urn:request:3b1f65f4-ee1d-4ebe-8c79-d5fdfb56aa02 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b1f65f4-ee1d-4ebe-8c79-d5fdfb56aa02', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:16:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:16:45,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:16:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:16:45,639 [285] [DEBUG] [app] Starting request: urn:request:071666c8-20b1-45a3-9e39-bc99a6d2447c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:45,639 [285] [DEBUG] [app] Ending request: urn:request:071666c8-20b1-45a3-9e39-bc99a6d2447c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:071666c8-20b1-45a3-9e39-bc99a6d2447c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:16:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:16:45,640 [284] [DEBUG] [app] Starting request: urn:request:6a09b312-b468-4634-8625-ecdda4667aee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:16:45,640 [284] [DEBUG] [app] Ending request: urn:request:6a09b312-b468-4634-8625-ecdda4667aee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a09b312-b468-4634-8625-ecdda4667aee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:16:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:16:45,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:16:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:16:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:45,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:16:45,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:16:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:45,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:16:45,660 [286] [DEBUG] [app] Ending request: urn:request:d1870465-5a49-47af-b49c-698a9530a58c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d1870465-5a49-47af-b49c-698a9530a58c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:16:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:16:45,661 [284] [DEBUG] [app] Ending request: urn:request:72db79d7-4ac0-48d8-8dd3-51419b7b1823 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:72db79d7-4ac0-48d8-8dd3-51419b7b1823', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:16:45,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:16:45,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:16:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:16:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.031)
securityworker stdout | 2024-04-22 04:16:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:16:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:14.067050+00:00 (in 27.000736 seconds)
securityworker stdout | 2024-04-22 04:16:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:17 UTC)" (scheduled at 2024-04-22 04:16:47.065745+00:00)
securityworker stdout | 2024-04-22 04:16:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:16:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:16:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:16:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:16:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:16:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:16:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:16:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:10.248668+00:00 (in 23.002261 seconds)
namespacegcworker stdout | 2024-04-22 04:16:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:47 UTC)" (scheduled at 2024-04-22 04:16:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:16:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:16:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 47, 246792), True, datetime.datetime(2024, 4, 22, 4, 16, 47, 246792), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:16:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:16:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:16:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:16:47,411 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:16:47,547 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:16:48,105 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:16:48,659 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:16:49,636 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:16:50,023 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:16:50,026 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:16:50,029 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:16:50,785 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:16:51,170 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:16:51,569 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:16:51,635 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:16:51,640 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:16:51,643 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:16:51,647 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:16:51,650 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:16:51,653 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:16:51,655 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:16:51,658 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:16:51,664 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:16:51,677 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:16:51,679 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:16:51,686 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:16:51,688 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:16:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:16:54.433683+00:00 (in 2.002214 seconds)
notificationworker stdout | 2024-04-22 04:16:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:02 UTC)" (scheduled at 2024-04-22 04:16:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:16:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:16:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 52, 431751), True, datetime.datetime(2024, 4, 22, 4, 16, 52, 431751), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:16:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:16:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:16:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:16:52,474 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:16:52,701 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:16:52,744 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:16:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:16:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:18.047111+00:00 (in 25.001056 seconds)
exportactionlogsworker stdout | 2024-04-22 04:16:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:53 UTC)" (scheduled at 2024-04-22 04:16:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:16:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:16:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 16, 53, 46325), True, datetime.datetime(2024, 4, 22, 4, 16, 53, 46325), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:16:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:16:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:16:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:16:53,614 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:16:53,616 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:16:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:16:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:02.430973+00:00 (in 7.996777 seconds)
notificationworker stdout | 2024-04-22 04:16:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:54 UTC)" (scheduled at 2024-04-22 04:16:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:16:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:16:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:17:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:16:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:16:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:12.806778+00:00 (in 14.999472 seconds)
quotatotalworker stdout | 2024-04-22 04:16:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:12 UTC)" (scheduled at 2024-04-22 04:16:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:16:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:16:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:16:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:16:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:16:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:28.130973+00:00 (in 29.999492 seconds)
buildlogsarchiver stdout | 2024-04-22 04:16:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:28 UTC)" (scheduled at 2024-04-22 04:16:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:16:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 16, 58, 131877), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:16:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:16:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:16:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:17:00,632 [286] [DEBUG] [app] Starting request: urn:request:95042eef-bd55-4b48-b29d-66bc137f0437 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:00,632 [282] [DEBUG] [app] Starting request: urn:request:d37587b0-7f1a-4306-a490-e474676cb386 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:17:00,635 [281] [DEBUG] [app] Starting request: urn:request:20ac6cc4-ba69-41e4-a8a0-4c9501b1b010 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:00,635 [283] [DEBUG] [app] Starting request: urn:request:498fe03d-d136-4084-a285-56c2151722b6 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:00,635 [281] [DEBUG] [app] Ending request: urn:request:20ac6cc4-ba69-41e4-a8a0-4c9501b1b010 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:20ac6cc4-ba69-41e4-a8a0-4c9501b1b010', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:00,635 [283] [DEBUG] [app] Ending request: urn:request:498fe03d-d136-4084-a285-56c2151722b6 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:498fe03d-d136-4084-a285-56c2151722b6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:17:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:17:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:00,638 [286] [DEBUG] [app] Starting request: urn:request:a76aa924-e7a4-4104-9c88-3a21c606bcb3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:00,638 [284] [DEBUG] [app] Starting request: urn:request:f46bbc43-359e-4b0e-8e06-7495aa811710 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:00,638 [286] [DEBUG] [app] Ending request: urn:request:a76aa924-e7a4-4104-9c88-3a21c606bcb3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a76aa924-e7a4-4104-9c88-3a21c606bcb3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:00,639 [284] [DEBUG] [app] Ending request: urn:request:f46bbc43-359e-4b0e-8e06-7495aa811710 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f46bbc43-359e-4b0e-8e06-7495aa811710', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:17:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:00,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:17:00,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:00,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:00,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:00,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:00,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:00,659 [286] [DEBUG] [app] Ending request: urn:request:95042eef-bd55-4b48-b29d-66bc137f0437 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95042eef-bd55-4b48-b29d-66bc137f0437', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:00,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:00,660 [282] [DEBUG] [app] Ending request: urn:request:d37587b0-7f1a-4306-a490-e474676cb386 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:d37587b0-7f1a-4306-a490-e474676cb386', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:00,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:17:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:00,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gcworker stdout | 2024-04-22 04:17:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:17:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:31.842666+00:00 (in 29.999458 seconds)
gcworker stdout | 2024-04-22 04:17:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:31 UTC)" (scheduled at 2024-04-22 04:17:01.842666+00:00)
gcworker stdout | 2024-04-22 04:17:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549821843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:17:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:17:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:17:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:17:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:17:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:09.033206+00:00 (in 6.997071 seconds)
repositorygcworker stdout | 2024-04-22 04:17:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:02 UTC)" (scheduled at 2024-04-22 04:17:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:17:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:17:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:17:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:12.430973+00:00 (in 9.999410 seconds)
notificationworker stdout | 2024-04-22 04:17:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:12 UTC)" (scheduled at 2024-04-22 04:17:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:17:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:17:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 2, 432004), True, datetime.datetime(2024, 4, 22, 4, 17, 2, 432004), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:17:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:17:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:17:05,848 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:17:07,192 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:17:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:17:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:02.035613+00:00 (in 53.001932 seconds)
repositorygcworker stdout | 2024-04-22 04:17:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:09 UTC)" (scheduled at 2024-04-22 04:17:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:17:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:17:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 9, 34158), True, datetime.datetime(2024, 4, 22, 4, 17, 9, 34158), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:17:09,060 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:17:09,060 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:17:09,061 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:17:09,673 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:17:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:17:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:40.238173+00:00 (in 29.999510 seconds)
autopruneworker stdout | 2024-04-22 04:17:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:40 UTC)" (scheduled at 2024-04-22 04:17:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:17:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759370247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:17:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:17:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:47.245913+00:00 (in 36.996780 seconds)
namespacegcworker stdout | 2024-04-22 04:17:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:10 UTC)" (scheduled at 2024-04-22 04:17:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:17:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:17:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:17:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:17:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:17:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:17:10,690 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:17:11,011 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:17:11,251 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:17:11,695 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:17:12,382 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:17:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:22.430973+00:00 (in 9.999495 seconds)
notificationworker stdout | 2024-04-22 04:17:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:22 UTC)" (scheduled at 2024-04-22 04:17:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:17:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:17:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 12, 431863), True, datetime.datetime(2024, 4, 22, 4, 17, 12, 431863), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:17:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:17:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:22 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:17:12,674 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:17:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:17:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:27.806778+00:00 (in 14.999445 seconds)
quotatotalworker stdout | 2024-04-22 04:17:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:27 UTC)" (scheduled at 2024-04-22 04:17:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:17:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:17:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:17:12,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:17:13,921 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:17:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:17:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:17.065745+00:00 (in 2.998154 seconds)
securityworker stdout | 2024-04-22 04:17:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:44 UTC)" (scheduled at 2024-04-22 04:17:14.067050+00:00)
securityworker stdout | 2024-04-22 04:17:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:17:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:17:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:17:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:17:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:17:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:17:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:17:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:17:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 12, 14, 77430), 1, 2])
securityworker stdout | 2024-04-22 04:17:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 12, 14, 77430), 1, 2])
securityworker stdout | 2024-04-22 04:17:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:17:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:17:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:17:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:17:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:17:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:17:14,893 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:17:15,632 [284] [DEBUG] [app] Starting request: urn:request:95613d2e-200d-41b6-a0ce-b2679d0073d5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:15,632 [282] [DEBUG] [app] Starting request: urn:request:f71532ae-c2e6-4725-8602-2ad402e99dae (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:17:15,635 [283] [DEBUG] [app] Starting request: urn:request:0d1abc79-aea1-4aa9-b88a-b6cd0e9efa5f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:15,635 [280] [DEBUG] [app] Starting request: urn:request:9e772810-8b9d-4600-8c2c-53d72132df38 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:15,635 [283] [DEBUG] [app] Ending request: urn:request:0d1abc79-aea1-4aa9-b88a-b6cd0e9efa5f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0d1abc79-aea1-4aa9-b88a-b6cd0e9efa5f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:15,635 [280] [DEBUG] [app] Ending request: urn:request:9e772810-8b9d-4600-8c2c-53d72132df38 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9e772810-8b9d-4600-8c2c-53d72132df38', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:17:15,635 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:17:15,635 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:15,635 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:15,635 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:15,638 [282] [DEBUG] [app] Starting request: urn:request:c2d9c889-ccd3-4566-ad4a-6f0140104b7d (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:15,638 [282] [DEBUG] [app] Ending request: urn:request:c2d9c889-ccd3-4566-ad4a-6f0140104b7d (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:c2d9c889-ccd3-4566-ad4a-6f0140104b7d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:15,638 [285] [DEBUG] [app] Starting request: urn:request:6122673a-ce97-4c40-8233-d544b74fec41 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:17:15,638 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:15,638 [285] [DEBUG] [app] Ending request: urn:request:6122673a-ce97-4c40-8233-d544b74fec41 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6122673a-ce97-4c40-8233-d544b74fec41', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:15,638 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:17:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:15,639 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:15,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:15,647 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:15,659 [284] [DEBUG] [app] Ending request: urn:request:95613d2e-200d-41b6-a0ce-b2679d0073d5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:95613d2e-200d-41b6-a0ce-b2679d0073d5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:15,659 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:15,659 [282] [DEBUG] [app] Ending request: urn:request:f71532ae-c2e6-4725-8602-2ad402e99dae (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f71532ae-c2e6-4725-8602-2ad402e99dae', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:15,659 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:17:15,659 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:17:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:17:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:44.067050+00:00 (in 27.000745 seconds)
securityworker stdout | 2024-04-22 04:17:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:47 UTC)" (scheduled at 2024-04-22 04:17:17.065745+00:00)
securityworker stdout | 2024-04-22 04:17:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:17:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:17:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:17:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:17:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:17:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:17:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 12, 17, 70467), 1, 2])
securityworker stdout | 2024-04-22 04:17:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 12, 17, 70467), 1, 2])
securityworker stdout | 2024-04-22 04:17:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:17,099 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:17:17,099 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:17:17,448 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:17:17,556 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:17:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:17:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:23.046900+00:00 (in 4.999296 seconds)
exportactionlogsworker stdout | 2024-04-22 04:17:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:18 UTC)" (scheduled at 2024-04-22 04:17:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:17:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:17:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:17:18,120 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:17:18,697 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:17:19,642 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:17:20,031 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:17:20,034 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:17:20,037 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:17:20,822 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:17:21,207 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:17:21,578 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:17:21,644 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:17:21,648 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:17:21,653 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:17:21,655 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:17:21,659 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:17:21,662 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:17:21,664 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:17:21,667 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:17:21,670 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-web stdout | 2024-04-22 04:17:21,685 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:17:21,687 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:17:21,693 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:17:21,695 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:17:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:32.430973+00:00 (in 9.999503 seconds)
notificationworker stdout | 2024-04-22 04:17:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:32 UTC)" (scheduled at 2024-04-22 04:17:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:17:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:17:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 22, 431894), True, datetime.datetime(2024, 4, 22, 4, 17, 22, 431894), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:17:22,448 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:17:22,448 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:22,448 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:17:22,510 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:17:22,737 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:17:22,768 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:17:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:17:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:53.045554+00:00 (in 29.998165 seconds)
exportactionlogsworker stdout | 2024-04-22 04:17:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:22:23 UTC)" (scheduled at 2024-04-22 04:17:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 04:17:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 17, 23, 47597), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:17:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 17, 23, 47597), True, datetime.datetime(2024, 4, 22, 4, 17, 23, 47597), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:17:23,065 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 17, 23, 47597), True, datetime.datetime(2024, 4, 22, 4, 17, 23, 47597), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 4, 17, 23, 47597), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:17:23,069 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:17:23,069 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:22:23 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:17:23,631 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:17:23,649 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:17:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:17:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:24.738365+00:00 (in 59.999482 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:17:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:24 UTC)" (scheduled at 2024-04-22 04:17:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:17:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:17:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:17:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:17:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:17:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:42.806778+00:00 (in 14.999506 seconds)
quotatotalworker stdout | 2024-04-22 04:17:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:42 UTC)" (scheduled at 2024-04-22 04:17:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:17:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:17:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:17:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:17:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:17:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:58.130973+00:00 (in 29.999464 seconds)
buildlogsarchiver stdout | 2024-04-22 04:17:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:58 UTC)" (scheduled at 2024-04-22 04:17:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:17:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 17, 28, 131913), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:17:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:17:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:17:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:17:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:31.933721+00:00 (in 2.002208 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:29 UTC)" (scheduled at 2024-04-22 04:17:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 29, 931958), True, datetime.datetime(2024, 4, 22, 4, 17, 29, 931958), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:17:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:17:30,632 [286] [DEBUG] [app] Starting request: urn:request:2bcedfd2-c3b3-41b0-a21c-9516b00b998b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:30,632 [282] [DEBUG] [app] Starting request: urn:request:7f23a460-b931-441b-8249-7fb1b28dc96b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:30,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:17:30,636 [283] [DEBUG] [app] Starting request: urn:request:76554d85-8e82-4d8e-b6c8-9bc687b7e764 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:30,636 [281] [DEBUG] [app] Starting request: urn:request:81a5a408-9536-4cb0-8c31-83a6a7dc84d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:30,637 [283] [DEBUG] [app] Ending request: urn:request:76554d85-8e82-4d8e-b6c8-9bc687b7e764 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76554d85-8e82-4d8e-b6c8-9bc687b7e764', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:30,637 [281] [DEBUG] [app] Ending request: urn:request:81a5a408-9536-4cb0-8c31-83a6a7dc84d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:81a5a408-9536-4cb0-8c31-83a6a7dc84d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:30,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:17:30,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:30,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:30,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:30,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:30,640 [284] [DEBUG] [app] Starting request: urn:request:44974f93-4d0c-4bdf-adad-b17f99229c93 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:30,640 [285] [DEBUG] [app] Starting request: urn:request:3395cce8-a004-486e-b11c-aa85c3337a6b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:30,641 [284] [DEBUG] [app] Ending request: urn:request:44974f93-4d0c-4bdf-adad-b17f99229c93 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:44974f93-4d0c-4bdf-adad-b17f99229c93', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:30,641 [285] [DEBUG] [app] Ending request: urn:request:3395cce8-a004-486e-b11c-aa85c3337a6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3395cce8-a004-486e-b11c-aa85c3337a6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:30,641 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:30,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:30,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:17:30,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:30,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:30,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:30,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:30,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:30,662 [282] [DEBUG] [app] Ending request: urn:request:7f23a460-b931-441b-8249-7fb1b28dc96b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7f23a460-b931-441b-8249-7fb1b28dc96b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:30,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:30,662 [286] [DEBUG] [app] Ending request: urn:request:2bcedfd2-c3b3-41b0-a21c-9516b00b998b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:2bcedfd2-c3b3-41b0-a21c-9516b00b998b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:30,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:30,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:17:30,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gcworker stdout | 2024-04-22 04:17:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:17:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:01.842666+00:00 (in 29.999523 seconds)
gcworker stdout | 2024-04-22 04:17:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:01 UTC)" (scheduled at 2024-04-22 04:17:31.842666+00:00)
gcworker stdout | 2024-04-22 04:17:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549851843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:17:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:17:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:17:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:17:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:17:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:29.931002+00:00 (in 57.996869 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:17:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:31 UTC)" (scheduled at 2024-04-22 04:17:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:17:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:17:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:17:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:38.433394+00:00 (in 6.001903 seconds)
notificationworker stdout | 2024-04-22 04:17:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:42 UTC)" (scheduled at 2024-04-22 04:17:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:17:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:17:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 32, 431881), True, datetime.datetime(2024, 4, 22, 4, 17, 32, 431881), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:17:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:17:32,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:32,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:17:35,868 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:17:37,202 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
notificationworker stdout | 2024-04-22 04:17:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:42.430973+00:00 (in 3.997107 seconds)
notificationworker stdout | 2024-04-22 04:17:38,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:22:38 UTC)" (scheduled at 2024-04-22 04:17:38.433394+00:00)
notificationworker stdout | 2024-04-22 04:17:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 17, 38, 434236), 'notification/%'])
notificationworker stdout | 2024-04-22 04:17:38,448 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 17, 38, 434236), True, datetime.datetime(2024, 4, 22, 4, 17, 38, 434236), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 04:17:38,453 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 17, 38, 434236), True, datetime.datetime(2024, 4, 22, 4, 17, 38, 434236), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 4, 17, 38, 434236), 'notification/%'])
notificationworker stdout | 2024-04-22 04:17:38,456 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:38,457 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:22:38 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:17:39,685 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:17:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:17:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:10.238173+00:00 (in 29.999444 seconds)
autopruneworker stdout | 2024-04-22 04:17:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:10 UTC)" (scheduled at 2024-04-22 04:17:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:17:40,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759400247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:17:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:17:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:17:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:17:40,726 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:17:41,048 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:17:41,288 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:17:41,711 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:17:42,390 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:17:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:52.430973+00:00 (in 9.999525 seconds)
notificationworker stdout | 2024-04-22 04:17:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:52 UTC)" (scheduled at 2024-04-22 04:17:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:17:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:17:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 42, 431884), True, datetime.datetime(2024, 4, 22, 4, 17, 42, 431884), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:17:42,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:17:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:17:52 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:17:42,710 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:17:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:17:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:57.806778+00:00 (in 14.999488 seconds)
quotatotalworker stdout | 2024-04-22 04:17:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:57 UTC)" (scheduled at 2024-04-22 04:17:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:17:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:17:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:17:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:17:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:17:43,957 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:17:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:17:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:47.065745+00:00 (in 2.998164 seconds)
securityworker stdout | 2024-04-22 04:17:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:14 UTC)" (scheduled at 2024-04-22 04:17:44.067050+00:00)
securityworker stdout | 2024-04-22 04:17:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:17:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:17:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:17:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:17:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:17:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:17:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:17:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:17:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 12, 44, 77501), 1, 2])
securityworker stdout | 2024-04-22 04:17:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 12, 44, 77501), 1, 2])
securityworker stdout | 2024-04-22 04:17:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:17:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:17:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:17:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:17:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:17:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:17:44,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:17:44,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:17:44,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:17:44,920 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:17:45,632 [285] [DEBUG] [app] Starting request: urn:request:f15eb59f-080c-4277-9f6d-5ee02a9a1b34 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:45,632 [286] [DEBUG] [app] Starting request: urn:request:945f1ca7-21d4-48a3-a3a9-7a9a413dadef (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:17:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:17:45,635 [275] [DEBUG] [app] Starting request: urn:request:b2049be4-02fe-4336-b22b-f7cf8f96728e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:45,636 [283] [DEBUG] [app] Starting request: urn:request:d6724a0c-5870-45a4-a1f4-44e5a7a00178 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:17:45,636 [275] [DEBUG] [app] Ending request: urn:request:b2049be4-02fe-4336-b22b-f7cf8f96728e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2049be4-02fe-4336-b22b-f7cf8f96728e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:45,636 [283] [DEBUG] [app] Ending request: urn:request:d6724a0c-5870-45a4-a1f4-44e5a7a00178 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d6724a0c-5870-45a4-a1f4-44e5a7a00178', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:17:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:17:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:45,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:17:45,639 [282] [DEBUG] [app] Starting request: urn:request:95fc82cb-73d5-4321-a03a-34fa3af8c4f5 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:45,639 [286] [DEBUG] [app] Starting request: urn:request:122304ed-9208-4428-9bfc-b8e7f3749c23 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:17:45,639 [282] [DEBUG] [app] Ending request: urn:request:95fc82cb-73d5-4321-a03a-34fa3af8c4f5 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:95fc82cb-73d5-4321-a03a-34fa3af8c4f5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:17:45,639 [286] [DEBUG] [app] Ending request: urn:request:122304ed-9208-4428-9bfc-b8e7f3749c23 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:122304ed-9208-4428-9bfc-b8e7f3749c23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:17:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:17:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:45,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:17:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:17:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:17:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:17:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:17:45,660 [285] [DEBUG] [app] Ending request: urn:request:f15eb59f-080c-4277-9f6d-5ee02a9a1b34 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f15eb59f-080c-4277-9f6d-5ee02a9a1b34', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:45,660 [286] [DEBUG] [app] Ending request: urn:request:945f1ca7-21d4-48a3-a3a9-7a9a413dadef (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:945f1ca7-21d4-48a3-a3a9-7a9a413dadef', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:17:45,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:17:45,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:17:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:17:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:17:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:17:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:17:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:14.067050+00:00 (in 27.000766 seconds)
securityworker stdout | 2024-04-22 04:17:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:17 UTC)" (scheduled at 2024-04-22 04:17:47.065745+00:00)
securityworker stdout | 2024-04-22 04:17:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:17:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:17:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:17:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:17:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:17:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:17:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:17:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:10.248668+00:00 (in 23.002244 seconds)
namespacegcworker stdout | 2024-04-22 04:17:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:47 UTC)" (scheduled at 2024-04-22 04:17:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:17:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:17:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 47, 246802), True, datetime.datetime(2024, 4, 22, 4, 17, 47, 246802), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:17:47,260 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:17:47,260 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:17:47,260 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:17:47,485 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:17:47,580 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:17:48,139 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:17:48,733 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:17:49,653 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:17:50,039 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:17:50,042 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:17:50,044 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:17:50,850 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:17:51,244 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:17:51,590 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:17:51,652 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:17:51,655 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:17:51,662 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:17:51,665 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:17:51,669 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:17:51,672 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:17:51,674 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-web stdout | 2024-04-22 04:17:51,678 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:17:51,680 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:17:51,692 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:17:51,694 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:17:51,699 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:17:51,702 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:17:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:17:54.433683+00:00 (in 2.002197 seconds)
notificationworker stdout | 2024-04-22 04:17:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:02 UTC)" (scheduled at 2024-04-22 04:17:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:17:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:17:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 52, 431761), True, datetime.datetime(2024, 4, 22, 4, 17, 52, 431761), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:17:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:17:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:17:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:17:52,518 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:17:52,774 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:17:52,803 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:17:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:17:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:18.047111+00:00 (in 25.000998 seconds)
exportactionlogsworker stdout | 2024-04-22 04:17:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:53 UTC)" (scheduled at 2024-04-22 04:17:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:17:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:17:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 17, 53, 46508), True, datetime.datetime(2024, 4, 22, 4, 17, 53, 46508), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:17:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:17:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:17:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:53 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:17:53,670 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:17:53,685 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:17:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:17:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:02.430973+00:00 (in 7.996795 seconds)
notificationworker stdout | 2024-04-22 04:17:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:54 UTC)" (scheduled at 2024-04-22 04:17:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:17:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:17:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:18:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:17:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:17:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:12.806778+00:00 (in 14.999484 seconds)
quotatotalworker stdout | 2024-04-22 04:17:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:12 UTC)" (scheduled at 2024-04-22 04:17:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:17:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:17:57,824 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:17:57,824 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:17:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:17:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:28.130973+00:00 (in 29.999456 seconds)
buildlogsarchiver stdout | 2024-04-22 04:17:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:28 UTC)" (scheduled at 2024-04-22 04:17:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:17:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 17, 58, 131887), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:17:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:17:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:17:58,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:18:00,632 [286] [DEBUG] [app] Starting request: urn:request:5d099089-a8fd-4ecd-81ac-579974cad911 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:00,632 [282] [DEBUG] [app] Starting request: urn:request:83d3ee30-e306-4ddc-8b24-f810bb8bc1a8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:18:00,635 [275] [DEBUG] [app] Starting request: urn:request:0550d448-6693-4418-b34d-fd84ce34657f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:00,635 [276] [DEBUG] [app] Starting request: urn:request:b59db160-2e24-4145-8b6b-adb6d0ae76a5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:00,635 [275] [DEBUG] [app] Ending request: urn:request:0550d448-6693-4418-b34d-fd84ce34657f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0550d448-6693-4418-b34d-fd84ce34657f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:00,635 [276] [DEBUG] [app] Ending request: urn:request:b59db160-2e24-4145-8b6b-adb6d0ae76a5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b59db160-2e24-4145-8b6b-adb6d0ae76a5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:18:00,636 [276] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:00,638 [285] [DEBUG] [app] Starting request: urn:request:32a5e431-c669-4c8b-bc95-9a63112565c1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:00,638 [284] [DEBUG] [app] Starting request: urn:request:6a9ced69-9b88-4c02-aaa4-032ffe94cc28 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:00,639 [285] [DEBUG] [app] Ending request: urn:request:32a5e431-c669-4c8b-bc95-9a63112565c1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:32a5e431-c669-4c8b-bc95-9a63112565c1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:18:00,639 [284] [DEBUG] [app] Ending request: urn:request:6a9ced69-9b88-4c02-aaa4-032ffe94cc28 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:6a9ced69-9b88-4c02-aaa4-032ffe94cc28', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:18:00,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:00,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:18:00,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:00,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:00,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:00,660 [282] [DEBUG] [app] Ending request: urn:request:83d3ee30-e306-4ddc-8b24-f810bb8bc1a8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:83d3ee30-e306-4ddc-8b24-f810bb8bc1a8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:00,660 [286] [DEBUG] [app] Ending request: urn:request:5d099089-a8fd-4ecd-81ac-579974cad911 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5d099089-a8fd-4ecd-81ac-579974cad911', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:00,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:00,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:18:00,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:18:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:18:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:31.842666+00:00 (in 29.999513 seconds)
gcworker stdout | 2024-04-22 04:18:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:31 UTC)" (scheduled at 2024-04-22 04:18:01.842666+00:00)
gcworker stdout | 2024-04-22 04:18:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549881843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:18:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:18:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:18:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:18:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:18:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:09.033206+00:00 (in 6.996997 seconds)
repositorygcworker stdout | 2024-04-22 04:18:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:02 UTC)" (scheduled at 2024-04-22 04:18:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:18:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:18:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:18:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:12.430973+00:00 (in 9.999439 seconds)
notificationworker stdout | 2024-04-22 04:18:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:12 UTC)" (scheduled at 2024-04-22 04:18:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:18:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:18:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 2, 431774), True, datetime.datetime(2024, 4, 22, 4, 18, 2, 431774), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:18:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:18:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:18:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:18:05,895 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:18:07,239 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:18:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:18:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:26.035364+00:00 (in 17.001652 seconds)
repositorygcworker stdout | 2024-04-22 04:18:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:09 UTC)" (scheduled at 2024-04-22 04:18:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:18:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:18:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 9, 34162), True, datetime.datetime(2024, 4, 22, 4, 18, 9, 34162), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:18:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:18:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:18:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:18:09,715 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:18:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:18:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:40.238173+00:00 (in 29.999435 seconds)
autopruneworker stdout | 2024-04-22 04:18:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:40 UTC)" (scheduled at 2024-04-22 04:18:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:18:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759430247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:18:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:18:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:47.245913+00:00 (in 36.996810 seconds)
namespacegcworker stdout | 2024-04-22 04:18:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:10 UTC)" (scheduled at 2024-04-22 04:18:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:18:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:18:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:18:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:18:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:18:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:18:10,763 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:18:11,084 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:18:11,316 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:18:11,747 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
teamsyncworker stdout | 2024-04-22 04:18:12,427 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:18:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:22.430973+00:00 (in 9.999551 seconds)
notificationworker stdout | 2024-04-22 04:18:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:22 UTC)" (scheduled at 2024-04-22 04:18:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:18:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:18:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 12, 431672), True, datetime.datetime(2024, 4, 22, 4, 18, 12, 431672), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:18:12,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:18:12,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:18:12,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:22 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:18:12,732 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:18:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:18:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:27.806778+00:00 (in 14.999448 seconds)
quotatotalworker stdout | 2024-04-22 04:18:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:27 UTC)" (scheduled at 2024-04-22 04:18:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:18:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:18:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:18:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:18:13,984 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:18:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:18:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:17.065745+00:00 (in 2.998157 seconds)
securityworker stdout | 2024-04-22 04:18:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:44 UTC)" (scheduled at 2024-04-22 04:18:14.067050+00:00)
securityworker stdout | 2024-04-22 04:18:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:18:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:18:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:18:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:18:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:18:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:18:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:18:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:18:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 13, 14, 78281), 1, 2])
securityworker stdout | 2024-04-22 04:18:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 13, 14, 78281), 1, 2])
securityworker stdout | 2024-04-22 04:18:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:18:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:18:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:18:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:18:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:18:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:18:14,956 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:18:15,631 [286] [DEBUG] [app] Starting request: urn:request:0d0580b1-67af-49f4-8391-b84240197c0a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:15,631 [282] [DEBUG] [app] Starting request: urn:request:bec85e89-7e9a-4c6a-bf81-e0646e236495 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:18:15,634 [275] [DEBUG] [app] Starting request: urn:request:2ef6704d-c3fa-4210-a7c5-36a72e84b12d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:15,635 [283] [DEBUG] [app] Starting request: urn:request:0340efb2-3344-4d5d-8909-f4b0b093724f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:15,635 [275] [DEBUG] [app] Ending request: urn:request:2ef6704d-c3fa-4210-a7c5-36a72e84b12d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2ef6704d-c3fa-4210-a7c5-36a72e84b12d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:15,635 [283] [DEBUG] [app] Ending request: urn:request:0340efb2-3344-4d5d-8909-f4b0b093724f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0340efb2-3344-4d5d-8909-f4b0b093724f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:15,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:18:15,635 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:15,635 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:15,635 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:15,638 [285] [DEBUG] [app] Starting request: urn:request:1dc4ea09-1ae5-44ab-a2b9-a02bfa729708 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:15,638 [282] [DEBUG] [app] Starting request: urn:request:f4def070-e432-4b3d-ac02-34a090781358 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:15,638 [282] [DEBUG] [app] Ending request: urn:request:f4def070-e432-4b3d-ac02-34a090781358 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f4def070-e432-4b3d-ac02-34a090781358', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:18:15,638 [285] [DEBUG] [app] Ending request: urn:request:1dc4ea09-1ae5-44ab-a2b9-a02bfa729708 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1dc4ea09-1ae5-44ab-a2b9-a02bfa729708', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:18:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:15,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:15,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:15,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:15,659 [286] [DEBUG] [app] Ending request: urn:request:0d0580b1-67af-49f4-8391-b84240197c0a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0d0580b1-67af-49f4-8391-b84240197c0a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:15,660 [282] [DEBUG] [app] Ending request: urn:request:bec85e89-7e9a-4c6a-bf81-e0646e236495 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bec85e89-7e9a-4c6a-bf81-e0646e236495', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:18:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:18:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
securityworker stdout | 2024-04-22 04:18:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:18:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:44.067050+00:00 (in 27.000769 seconds)
securityworker stdout | 2024-04-22 04:18:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:47 UTC)" (scheduled at 2024-04-22 04:18:17.065745+00:00)
securityworker stdout | 2024-04-22 04:18:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:18:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:18:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:18:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:18:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:18:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:18:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 13, 17, 70673), 1, 2])
securityworker stdout | 2024-04-22 04:18:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 13, 17, 70673), 1, 2])
securityworker stdout | 2024-04-22 04:18:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:18:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:18:17,521 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:18:17,615 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:18:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:18:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:53.045554+00:00 (in 34.997985 seconds)
exportactionlogsworker stdout | 2024-04-22 04:18:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:18 UTC)" (scheduled at 2024-04-22 04:18:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:18:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:18:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:18:18,175 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:18:18,770 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:18:19,661 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:18:20,047 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:18:20,050 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:18:20,052 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:18:20,875 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:18:21,276 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:18:21,601 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:18:21,659 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:18:21,661 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:18:21,670 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:18:21,674 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:18:21,678 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:18:21,681 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:18:21,684 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:18:21,687 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:18:21,690 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:18:21,698 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:18:21,701 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:18:21,706 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:18:21,708 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:18:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:32.430973+00:00 (in 9.999518 seconds)
notificationworker stdout | 2024-04-22 04:18:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:32 UTC)" (scheduled at 2024-04-22 04:18:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:18:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:18:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 22, 431734), True, datetime.datetime(2024, 4, 22, 4, 18, 22, 431734), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:18:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:18:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:18:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:18:22,554 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:18:22,809 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:18:22,814 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:18:23,706 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:18:23,708 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:18:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:18:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:24.738365+00:00 (in 59.999470 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:18:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:24 UTC)" (scheduled at 2024-04-22 04:18:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:18:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:18:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:18:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:18:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:18:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:02.035613+00:00 (in 35.999809 seconds)
repositorygcworker stdout | 2024-04-22 04:18:26,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:23:26 UTC)" (scheduled at 2024-04-22 04:18:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 04:18:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 18, 26, 36192), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:18:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 18, 26, 36192), True, datetime.datetime(2024, 4, 22, 4, 18, 26, 36192), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:18:26,055 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 18, 26, 36192), True, datetime.datetime(2024, 4, 22, 4, 18, 26, 36192), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 4, 18, 26, 36192), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:18:26,059 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:18:26,059 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:23:26 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:18:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:18:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:42.806778+00:00 (in 14.999412 seconds)
quotatotalworker stdout | 2024-04-22 04:18:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:42 UTC)" (scheduled at 2024-04-22 04:18:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:18:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:18:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:18:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:18:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:18:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:58.130973+00:00 (in 29.999544 seconds)
buildlogsarchiver stdout | 2024-04-22 04:18:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:58 UTC)" (scheduled at 2024-04-22 04:18:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:18:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 18, 28, 131771), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:18:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:18:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:18:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:18:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:30.933445+00:00 (in 1.001945 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:29 UTC)" (scheduled at 2024-04-22 04:18:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 29, 931768), True, datetime.datetime(2024, 4, 22, 4, 18, 29, 931768), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:18:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:18:30,631 [286] [DEBUG] [app] Starting request: urn:request:0ec6eb8a-f6fc-4dc1-8804-a0efcd0677a4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:30,631 [282] [DEBUG] [app] Starting request: urn:request:90283dbf-f948-475d-877a-621f8ab9361c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:18:30,635 [275] [DEBUG] [app] Starting request: urn:request:ef149d85-3bdb-4486-bafa-a8725e8549d3 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:30,635 [281] [DEBUG] [app] Starting request: urn:request:96b8a1b7-c947-43e3-b3f3-1bbd64bbb438 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:30,635 [275] [DEBUG] [app] Ending request: urn:request:ef149d85-3bdb-4486-bafa-a8725e8549d3 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef149d85-3bdb-4486-bafa-a8725e8549d3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:30,635 [281] [DEBUG] [app] Ending request: urn:request:96b8a1b7-c947-43e3-b3f3-1bbd64bbb438 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:96b8a1b7-c947-43e3-b3f3-1bbd64bbb438', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:30,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:18:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:18:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:30,638 [286] [DEBUG] [app] Starting request: urn:request:397a8fb6-e4d7-4b42-b068-f7dda10e3647 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:30,638 [286] [DEBUG] [app] Ending request: urn:request:397a8fb6-e4d7-4b42-b068-f7dda10e3647 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:397a8fb6-e4d7-4b42-b068-f7dda10e3647', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:18:30,639 [285] [DEBUG] [app] Starting request: urn:request:5ce4c504-537d-4846-8f18-6b26bb2ab2b3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:18:30,639 [285] [DEBUG] [app] Ending request: urn:request:5ce4c504-537d-4846-8f18-6b26bb2ab2b3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ce4c504-537d-4846-8f18-6b26bb2ab2b3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:18:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:30,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:18:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:30,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:30,660 [282] [DEBUG] [app] Ending request: urn:request:90283dbf-f948-475d-877a-621f8ab9361c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:90283dbf-f948-475d-877a-621f8ab9361c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:30,660 [286] [DEBUG] [app] Ending request: urn:request:0ec6eb8a-f6fc-4dc1-8804-a0efcd0677a4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0ec6eb8a-f6fc-4dc1-8804-a0efcd0677a4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:30,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:18:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:31.933721+00:00 (in 0.999777 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:23:30 UTC)" (scheduled at 2024-04-22 04:18:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 18, 30, 934191), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,948 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 18, 30, 934191), True, datetime.datetime(2024, 4, 22, 4, 18, 30, 934191), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,953 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 18, 30, 934191), True, datetime.datetime(2024, 4, 22, 4, 18, 30, 934191), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 4, 18, 30, 934191), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,956 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:18:30,956 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:23:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:18:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:18:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:01.842666+00:00 (in 29.999494 seconds)
gcworker stdout | 2024-04-22 04:18:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:01 UTC)" (scheduled at 2024-04-22 04:18:31.842666+00:00)
gcworker stdout | 2024-04-22 04:18:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549911843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:18:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:18:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:18:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:18:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:18:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:29.931002+00:00 (in 57.996790 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:18:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:31 UTC)" (scheduled at 2024-04-22 04:18:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:18:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:18:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:18:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:42.430973+00:00 (in 9.999531 seconds)
notificationworker stdout | 2024-04-22 04:18:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:42 UTC)" (scheduled at 2024-04-22 04:18:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:18:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:18:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 32, 431687), True, datetime.datetime(2024, 4, 22, 4, 18, 32, 431687), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:18:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:18:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:18:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:18:35,915 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:18:37,246 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:18:39,733 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:18:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:18:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:10.238173+00:00 (in 29.999478 seconds)
autopruneworker stdout | 2024-04-22 04:18:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:10 UTC)" (scheduled at 2024-04-22 04:18:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:18:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759460247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:18:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:18:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:18:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:18:40,779 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:18:41,117 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:18:41,352 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:18:41,758 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:18:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:52.430973+00:00 (in 9.999500 seconds)
notificationworker stdout | 2024-04-22 04:18:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:52 UTC)" (scheduled at 2024-04-22 04:18:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:18:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:18:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 42, 431741), True, datetime.datetime(2024, 4, 22, 4, 18, 42, 431741), 0, 'notification/%', 50, 1, 0])
teamsyncworker stdout | 2024-04-22 04:18:42,445 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
notificationworker stdout | 2024-04-22 04:18:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:18:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:18:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:18:52 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:18:42,767 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:18:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:18:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:57.806778+00:00 (in 14.999438 seconds)
quotatotalworker stdout | 2024-04-22 04:18:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:57 UTC)" (scheduled at 2024-04-22 04:18:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:18:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:18:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:18:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:18:57 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:18:44,022 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:18:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:18:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:47.065745+00:00 (in 2.998144 seconds)
securityworker stdout | 2024-04-22 04:18:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:14 UTC)" (scheduled at 2024-04-22 04:18:44.067050+00:00)
securityworker stdout | 2024-04-22 04:18:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:18:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:18:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:18:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:18:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:18:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:18:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:18:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:18:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 13, 44, 77703), 1, 2])
securityworker stdout | 2024-04-22 04:18:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 13, 44, 77703), 1, 2])
securityworker stdout | 2024-04-22 04:18:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:18:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:18:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:18:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:18:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:18:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:18:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:18:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:18:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:18:44,993 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:18:45,632 [286] [DEBUG] [app] Starting request: urn:request:f0608ca3-a9f2-4e2f-a3a1-77c86e10d8cc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:45,632 [282] [DEBUG] [app] Starting request: urn:request:048d0eba-9816-4b69-a174-3b989219ed6e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:18:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:18:45,635 [279] [DEBUG] [app] Starting request: urn:request:e93d8216-860f-4504-b98d-9c6862b4194b (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:45,635 [275] [DEBUG] [app] Starting request: urn:request:4412c62d-ee1a-4f5a-9e13-b7af30b0f814 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:18:45,636 [279] [DEBUG] [app] Ending request: urn:request:e93d8216-860f-4504-b98d-9c6862b4194b (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e93d8216-860f-4504-b98d-9c6862b4194b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:18:45,636 [275] [DEBUG] [app] Ending request: urn:request:4412c62d-ee1a-4f5a-9e13-b7af30b0f814 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:4412c62d-ee1a-4f5a-9e13-b7af30b0f814', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:18:45,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:18:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:18:45,639 [282] [DEBUG] [app] Starting request: urn:request:3de07f6e-2b20-43fe-aa8f-579c8ab4b2c0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:45,639 [282] [DEBUG] [app] Ending request: urn:request:3de07f6e-2b20-43fe-aa8f-579c8ab4b2c0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3de07f6e-2b20-43fe-aa8f-579c8ab4b2c0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:18:45,639 [285] [DEBUG] [app] Starting request: urn:request:e18edc6a-6289-4540-975a-2054818467ca (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:18:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:18:45,639 [285] [DEBUG] [app] Ending request: urn:request:e18edc6a-6289-4540-975a-2054818467ca (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e18edc6a-6289-4540-975a-2054818467ca', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:18:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:45,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:18:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:18:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:18:45,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:18:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:18:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:18:45,660 [282] [DEBUG] [app] Ending request: urn:request:048d0eba-9816-4b69-a174-3b989219ed6e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:048d0eba-9816-4b69-a174-3b989219ed6e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:45,660 [286] [DEBUG] [app] Ending request: urn:request:f0608ca3-a9f2-4e2f-a3a1-77c86e10d8cc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f0608ca3-a9f2-4e2f-a3a1-77c86e10d8cc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:18:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:18:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:18:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:18:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:18:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:18:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:18:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:14.067050+00:00 (in 27.000731 seconds)
securityworker stdout | 2024-04-22 04:18:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:17 UTC)" (scheduled at 2024-04-22 04:18:47.065745+00:00)
securityworker stdout | 2024-04-22 04:18:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:18:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:18:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:18:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:18:47,094 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:18:47,094 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:18:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:18:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:10.248668+00:00 (in 23.002283 seconds)
namespacegcworker stdout | 2024-04-22 04:18:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:47 UTC)" (scheduled at 2024-04-22 04:18:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:18:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:18:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 47, 246815), True, datetime.datetime(2024, 4, 22, 4, 18, 47, 246815), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:18:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:18:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:18:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:18:47,553 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:18:47,636 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:18:48,212 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:18:48,808 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:18:49,677 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:18:50,055 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:18:50,058 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:18:50,061 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:18:50,912 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:18:51,311 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:18:51,613 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:18:51,671 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:18:51,673 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:18:51,682 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:18:51,684 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:18:51,690 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:18:51,693 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:18:51,695 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:18:51,698 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:18:51,700 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:18:51,705 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:18:51,709 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:18:51,713 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:18:51,716 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:18:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:18:54.433683+00:00 (in 2.002221 seconds)
notificationworker stdout | 2024-04-22 04:18:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:02 UTC)" (scheduled at 2024-04-22 04:18:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:18:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:18:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 52, 431725), True, datetime.datetime(2024, 4, 22, 4, 18, 52, 431725), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:18:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:18:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:18:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:18:52,566 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:18:52,817 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:18:52,848 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:18:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:18:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:18.047111+00:00 (in 25.001080 seconds)
exportactionlogsworker stdout | 2024-04-22 04:18:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:53 UTC)" (scheduled at 2024-04-22 04:18:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:18:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:18:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 18, 53, 46441), True, datetime.datetime(2024, 4, 22, 4, 18, 53, 46441), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:18:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:18:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:18:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:53 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:18:53,742 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:18:53,745 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:18:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:18:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:02.430973+00:00 (in 7.996786 seconds)
notificationworker stdout | 2024-04-22 04:18:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:54 UTC)" (scheduled at 2024-04-22 04:18:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:18:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:18:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:19:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:18:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:18:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:12.806778+00:00 (in 14.999443 seconds)
quotatotalworker stdout | 2024-04-22 04:18:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:12 UTC)" (scheduled at 2024-04-22 04:18:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:18:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:18:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:18:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:18:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:18:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:28.130973+00:00 (in 29.999524 seconds)
buildlogsarchiver stdout | 2024-04-22 04:18:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:28 UTC)" (scheduled at 2024-04-22 04:18:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:18:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 18, 58, 131871), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:18:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:18:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:18:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:19:00,632 [286] [DEBUG] [app] Starting request: urn:request:5eaeb0ea-2911-4b6a-82bd-2f1805a48886 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:00,632 [282] [DEBUG] [app] Starting request: urn:request:e4e4b561-9a4a-4fa2-b627-6843e2f20d99 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:19:00,636 [275] [DEBUG] [app] Starting request: urn:request:900d5f3e-f204-463e-bfbd-399730e67770 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:00,636 [275] [DEBUG] [app] Ending request: urn:request:900d5f3e-f204-463e-bfbd-399730e67770 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:900d5f3e-f204-463e-bfbd-399730e67770', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:19:00,636 [279] [DEBUG] [app] Starting request: urn:request:afa2095d-9fc5-4649-9f20-2599bbd400eb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:19:00,637 [279] [DEBUG] [app] Ending request: urn:request:afa2095d-9fc5-4649-9f20-2599bbd400eb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:afa2095d-9fc5-4649-9f20-2599bbd400eb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:19:00,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:19:00,637 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:00,639 [286] [DEBUG] [app] Starting request: urn:request:e8e78ec7-1ebe-4b70-b014-e36bdd4bb4d7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:19:00,640 [286] [DEBUG] [app] Ending request: urn:request:e8e78ec7-1ebe-4b70-b014-e36bdd4bb4d7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e8e78ec7-1ebe-4b70-b014-e36bdd4bb4d7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:00,640 [285] [DEBUG] [app] Starting request: urn:request:0912b120-fd6a-4c01-bd3e-e906c4011925 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:19:00,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:19:00,640 [285] [DEBUG] [app] Ending request: urn:request:0912b120-fd6a-4c01-bd3e-e906c4011925 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0912b120-fd6a-4c01-bd3e-e906c4011925', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:19:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:00,661 [282] [DEBUG] [app] Ending request: urn:request:e4e4b561-9a4a-4fa2-b627-6843e2f20d99 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e4e4b561-9a4a-4fa2-b627-6843e2f20d99', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:00,661 [286] [DEBUG] [app] Ending request: urn:request:5eaeb0ea-2911-4b6a-82bd-2f1805a48886 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5eaeb0ea-2911-4b6a-82bd-2f1805a48886', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:19:00,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:19:00,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:19:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:19:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:19:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:31.842666+00:00 (in 29.999544 seconds)
gcworker stdout | 2024-04-22 04:19:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:31 UTC)" (scheduled at 2024-04-22 04:19:01.842666+00:00)
gcworker stdout | 2024-04-22 04:19:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549941843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:19:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:19:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:19:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:19:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:19:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:09.033206+00:00 (in 6.997158 seconds)
repositorygcworker stdout | 2024-04-22 04:19:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:02 UTC)" (scheduled at 2024-04-22 04:19:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:19:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:19:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:19:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:12.430973+00:00 (in 9.999542 seconds)
notificationworker stdout | 2024-04-22 04:19:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:12 UTC)" (scheduled at 2024-04-22 04:19:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:19:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:19:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 2, 431706), True, datetime.datetime(2024, 4, 22, 4, 19, 2, 431706), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:19:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:19:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:19:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:19:05,929 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:19:07,270 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:19:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:19:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:02.035613+00:00 (in 53.001879 seconds)
repositorygcworker stdout | 2024-04-22 04:19:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:09 UTC)" (scheduled at 2024-04-22 04:19:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:19:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:19:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 9, 34183), True, datetime.datetime(2024, 4, 22, 4, 19, 9, 34183), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:19:09,050 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:19:09,050 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:19:09,050 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:19:09,769 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:19:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:19:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:40.238173+00:00 (in 29.999522 seconds)
autopruneworker stdout | 2024-04-22 04:19:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:40 UTC)" (scheduled at 2024-04-22 04:19:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:19:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759490247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:19:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:19:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:47.245913+00:00 (in 36.996734 seconds)
namespacegcworker stdout | 2024-04-22 04:19:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:10 UTC)" (scheduled at 2024-04-22 04:19:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:19:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:19:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:19:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:19:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:19:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:19:10,815 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:19:11,152 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:19:11,370 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:19:11,770 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:19:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:22.430973+00:00 (in 9.999436 seconds)
notificationworker stdout | 2024-04-22 04:19:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:22 UTC)" (scheduled at 2024-04-22 04:19:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:19:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:19:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 12, 431902), True, datetime.datetime(2024, 4, 22, 4, 19, 12, 431902), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:19:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:19:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:19:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:19:12,461 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:19:12,803 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:19:12,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:19:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:27.806778+00:00 (in 14.999473 seconds)
quotatotalworker stdout | 2024-04-22 04:19:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:27 UTC)" (scheduled at 2024-04-22 04:19:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:19:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:19:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:19:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:27 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:19:14,047 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:19:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:19:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:17.065745+00:00 (in 2.998201 seconds)
securityworker stdout | 2024-04-22 04:19:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:44 UTC)" (scheduled at 2024-04-22 04:19:14.067050+00:00)
securityworker stdout | 2024-04-22 04:19:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:19:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:19:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:19:14,071 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:19:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:19:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:19:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:19:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:19:14,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 14, 14, 76751), 1, 2])
securityworker stdout | 2024-04-22 04:19:14,097 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 14, 14, 76751), 1, 2])
securityworker stdout | 2024-04-22 04:19:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:14,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:14,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:19:14,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:19:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:19:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:19:14,103 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:19:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:44 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:19:15,029 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:19:15,632 [282] [DEBUG] [app] Starting request: urn:request:aae022fa-0ba5-4e3d-b585-bf8bd7cd1e83 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:15,632 [284] [DEBUG] [app] Starting request: urn:request:96884b7e-dd3f-45ba-b2e7-1d463ee2caa0 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:15,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:19:15,635 [281] [DEBUG] [app] Starting request: urn:request:621cd050-b9a9-4cc8-a7ee-d1715d11b292 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:15,635 [275] [DEBUG] [app] Starting request: urn:request:b2922ad4-6802-4dcd-8ba9-41ad2a5205ff (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:15,635 [281] [DEBUG] [app] Ending request: urn:request:621cd050-b9a9-4cc8-a7ee-d1715d11b292 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:621cd050-b9a9-4cc8-a7ee-d1715d11b292', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:19:15,636 [275] [DEBUG] [app] Ending request: urn:request:b2922ad4-6802-4dcd-8ba9-41ad2a5205ff (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b2922ad4-6802-4dcd-8ba9-41ad2a5205ff', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:19:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:19:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:15,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:19:15,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:15,639 [285] [DEBUG] [app] Starting request: urn:request:62ffb62c-8abe-48fa-a89a-8565847c29c9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:19:15,639 [285] [DEBUG] [app] Ending request: urn:request:62ffb62c-8abe-48fa-a89a-8565847c29c9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:62ffb62c-8abe-48fa-a89a-8565847c29c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:15,639 [284] [DEBUG] [app] Starting request: urn:request:07d8ec5d-06a7-4bd1-98a6-a31c6b67c3b7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:19:15,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:15,639 [284] [DEBUG] [app] Ending request: urn:request:07d8ec5d-06a7-4bd1-98a6-a31c6b67c3b7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:07d8ec5d-06a7-4bd1-98a6-a31c6b67c3b7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:19:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:15,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:15,660 [284] [DEBUG] [app] Ending request: urn:request:96884b7e-dd3f-45ba-b2e7-1d463ee2caa0 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:96884b7e-dd3f-45ba-b2e7-1d463ee2caa0', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:19:15,660 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:19:15,661 [282] [DEBUG] [app] Ending request: urn:request:aae022fa-0ba5-4e3d-b585-bf8bd7cd1e83 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:aae022fa-0ba5-4e3d-b585-bf8bd7cd1e83', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:19:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:19:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:19:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:44.067050+00:00 (in 27.000739 seconds)
securityworker stdout | 2024-04-22 04:19:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:47 UTC)" (scheduled at 2024-04-22 04:19:17.065745+00:00)
securityworker stdout | 2024-04-22 04:19:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:19:17,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:19:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:19:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:19:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:19:17,087 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:19:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:17,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 14, 17, 70708), 1, 2])
securityworker stdout | 2024-04-22 04:19:17,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:17,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 14, 17, 70708), 1, 2])
securityworker stdout | 2024-04-22 04:19:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:19:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:19:17,574 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:19:17,671 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:19:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:19:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:53.045554+00:00 (in 34.997932 seconds)
exportactionlogsworker stdout | 2024-04-22 04:19:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:18 UTC)" (scheduled at 2024-04-22 04:19:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:19:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:19:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:19:18,248 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:19:18,844 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:19:19,711 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:19:20,061 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:19:20,066 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:19:20,068 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:19:20,938 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:19:21,347 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:19:21,624 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:19:21,682 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:19:21,684 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:19:21,694 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:19:21,697 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:19:21,700 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-web stdout | 2024-04-22 04:19:21,702 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:19:21,705 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:19:21,708 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:19:21,712 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:19:21,715 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:19:21,718 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:19:21,720 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:19:21,723 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:19:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:32.430973+00:00 (in 9.999474 seconds)
notificationworker stdout | 2024-04-22 04:19:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:32 UTC)" (scheduled at 2024-04-22 04:19:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:19:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:19:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 22, 431791), True, datetime.datetime(2024, 4, 22, 4, 19, 22, 431791), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:19:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:19:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:19:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:19:22,603 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:19:22,854 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:19:22,857 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
quotatotalworker stdout | 2024-04-22 04:19:23,767 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:19:23,771 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:19:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:19:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:24.738365+00:00 (in 59.999462 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:19:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:24 UTC)" (scheduled at 2024-04-22 04:19:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:19:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:19:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:19:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:19:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:19:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:42.806778+00:00 (in 14.999418 seconds)
quotatotalworker stdout | 2024-04-22 04:19:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:42 UTC)" (scheduled at 2024-04-22 04:19:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:19:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:19:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:19:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:19:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:19:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:58.130973+00:00 (in 29.999460 seconds)
buildlogsarchiver stdout | 2024-04-22 04:19:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:58 UTC)" (scheduled at 2024-04-22 04:19:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:19:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 19, 28, 131917), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:19:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:19:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:19:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:19:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:31.933721+00:00 (in 2.002204 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:29 UTC)" (scheduled at 2024-04-22 04:19:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 29, 931783), True, datetime.datetime(2024, 4, 22, 4, 19, 29, 931783), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:19:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:19:30,632 [282] [DEBUG] [app] Starting request: urn:request:6fab66b1-3226-4416-99a8-121cd0213a3c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:30,632 [286] [DEBUG] [app] Starting request: urn:request:cec073db-4c98-4d51-9675-67704ae41581 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:19:30,635 [281] [DEBUG] [app] Starting request: urn:request:03bc3e2c-6eaf-44c9-bca8-11b6f52d22a2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:30,635 [280] [DEBUG] [app] Starting request: urn:request:043cf977-d40f-4306-84c0-b99c0d6f28b9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:30,636 [280] [DEBUG] [app] Ending request: urn:request:043cf977-d40f-4306-84c0-b99c0d6f28b9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:043cf977-d40f-4306-84c0-b99c0d6f28b9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:19:30,636 [281] [DEBUG] [app] Ending request: urn:request:03bc3e2c-6eaf-44c9-bca8-11b6f52d22a2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:03bc3e2c-6eaf-44c9-bca8-11b6f52d22a2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:19:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:19:30,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:19:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:30,638 [286] [DEBUG] [app] Starting request: urn:request:0ce67a1a-6531-464a-9031-5f14bd0daf78 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:19:30,639 [286] [DEBUG] [app] Ending request: urn:request:0ce67a1a-6531-464a-9031-5f14bd0daf78 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0ce67a1a-6531-464a-9031-5f14bd0daf78', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:30,639 [282] [DEBUG] [app] Starting request: urn:request:ba38f648-d1a0-4cd3-ba3d-6e12b654d3d1 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:19:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:19:30,639 [282] [DEBUG] [app] Ending request: urn:request:ba38f648-d1a0-4cd3-ba3d-6e12b654d3d1 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ba38f648-d1a0-4cd3-ba3d-6e12b654d3d1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:19:30,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:30,660 [282] [DEBUG] [app] Ending request: urn:request:6fab66b1-3226-4416-99a8-121cd0213a3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6fab66b1-3226-4416-99a8-121cd0213a3c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:19:30,660 [286] [DEBUG] [app] Ending request: urn:request:cec073db-4c98-4d51-9675-67704ae41581 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cec073db-4c98-4d51-9675-67704ae41581', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:19:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:19:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:19:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:19:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:01.842666+00:00 (in 29.999501 seconds)
gcworker stdout | 2024-04-22 04:19:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:01 UTC)" (scheduled at 2024-04-22 04:19:31.842666+00:00)
gcworker stdout | 2024-04-22 04:19:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712549971843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:19:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:19:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:19:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:19:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:19:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:29.931002+00:00 (in 57.996782 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:19:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:31 UTC)" (scheduled at 2024-04-22 04:19:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:19:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:19:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:19:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:42.430973+00:00 (in 9.999452 seconds)
notificationworker stdout | 2024-04-22 04:19:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:42 UTC)" (scheduled at 2024-04-22 04:19:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:19:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:19:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 32, 431843), True, datetime.datetime(2024, 4, 22, 4, 19, 32, 431843), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:19:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:19:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:19:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:19:35,966 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:19:37,304 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:19:39,804 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:19:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:19:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:10.238173+00:00 (in 29.999538 seconds)
autopruneworker stdout | 2024-04-22 04:19:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:10 UTC)" (scheduled at 2024-04-22 04:19:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:19:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759520247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:19:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:19:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:19:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:19:40,841 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:19:41,188 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:19:41,390 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:19:41,804 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:19:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:52.430973+00:00 (in 9.999442 seconds)
notificationworker stdout | 2024-04-22 04:19:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:52 UTC)" (scheduled at 2024-04-22 04:19:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:19:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:19:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 42, 431828), True, datetime.datetime(2024, 4, 22, 4, 19, 42, 431828), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:19:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:19:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:19:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:19:52 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:19:42,479 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:19:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:19:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:57.806778+00:00 (in 14.999395 seconds)
quotatotalworker stdout | 2024-04-22 04:19:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:57 UTC)" (scheduled at 2024-04-22 04:19:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:19:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
storagereplication stdout | 2024-04-22 04:19:42,813 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
quotatotalworker stdout | 2024-04-22 04:19:42,828 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:19:42,828 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:19:57 UTC)" executed successfully
securityworker stdout | 2024-04-22 04:19:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:19:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:47.065745+00:00 (in 2.998175 seconds)
securityworker stdout | 2024-04-22 04:19:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:14 UTC)" (scheduled at 2024-04-22 04:19:44.067050+00:00)
securityworker stdout | 2024-04-22 04:19:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:19:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:19:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:19:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:19:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:19:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
servicekey stdout | 2024-04-22 04:19:44,077 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
securityworker stdout | 2024-04-22 04:19:44,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:19:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:19:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:44,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 14, 44, 79140), 1, 2])
securityworker stdout | 2024-04-22 04:19:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 14, 44, 79140), 1, 2])
securityworker stdout | 2024-04-22 04:19:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:19:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:19:44,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:44,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:19:44,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:19:44,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:19:44,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:19:44,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:19:44,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:19:44,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:14 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:19:45,059 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:19:45,632 [286] [DEBUG] [app] Starting request: urn:request:0fba3c0e-ca61-4296-9788-87a1d3586f8e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:45,632 [285] [DEBUG] [app] Starting request: urn:request:6654e066-7c2f-47fe-a633-9f228ec843fb (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:19:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:19:45,636 [275] [DEBUG] [app] Starting request: urn:request:9215500c-dfb8-49b5-8e08-ec3e01b33f81 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:45,636 [275] [DEBUG] [app] Ending request: urn:request:9215500c-dfb8-49b5-8e08-ec3e01b33f81 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9215500c-dfb8-49b5-8e08-ec3e01b33f81', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:19:45,636 [281] [DEBUG] [app] Starting request: urn:request:9a135e04-9792-42a6-bd3c-c507dfbd4f4d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:19:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:19:45,637 [281] [DEBUG] [app] Ending request: urn:request:9a135e04-9792-42a6-bd3c-c507dfbd4f4d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9a135e04-9792-42a6-bd3c-c507dfbd4f4d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:19:45,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:19:45,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:45,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:45,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:19:45,639 [282] [DEBUG] [app] Starting request: urn:request:af3a4f4f-7e11-42f5-9be5-fa9080ed9d6b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:19:45,640 [282] [DEBUG] [app] Ending request: urn:request:af3a4f4f-7e11-42f5-9be5-fa9080ed9d6b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af3a4f4f-7e11-42f5-9be5-fa9080ed9d6b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:19:45,640 [285] [DEBUG] [app] Starting request: urn:request:065e3321-94f7-4fac-b8c1-e53dfbb9c994 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:19:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:45,640 [285] [DEBUG] [app] Ending request: urn:request:065e3321-94f7-4fac-b8c1-e53dfbb9c994 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:065e3321-94f7-4fac-b8c1-e53dfbb9c994', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:19:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:19:45,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:19:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:19:45,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:45,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:19:45,641 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:45,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:19:45,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:45,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:19:45,661 [286] [DEBUG] [app] Ending request: urn:request:0fba3c0e-ca61-4296-9788-87a1d3586f8e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:0fba3c0e-ca61-4296-9788-87a1d3586f8e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:45,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:19:45,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:19:45,661 [285] [DEBUG] [app] Ending request: urn:request:6654e066-7c2f-47fe-a633-9f228ec843fb (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:6654e066-7c2f-47fe-a633-9f228ec843fb', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:19:45,662 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:19:45,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:19:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:19:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:19:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:19:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:14.067050+00:00 (in 27.000690 seconds)
securityworker stdout | 2024-04-22 04:19:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:17 UTC)" (scheduled at 2024-04-22 04:19:47.065745+00:00)
securityworker stdout | 2024-04-22 04:19:47,067 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:19:47,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:19:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:19:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:19:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:19:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:19:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:19:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:10.248668+00:00 (in 23.002271 seconds)
namespacegcworker stdout | 2024-04-22 04:19:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:47 UTC)" (scheduled at 2024-04-22 04:19:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:19:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:19:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 47, 246631), True, datetime.datetime(2024, 4, 22, 4, 19, 47, 246631), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:19:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:19:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:19:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:19:47,597 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:19:47,708 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:19:48,282 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:19:48,865 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:19:49,747 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:19:50,069 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:19:50,074 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:19:50,077 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:19:50,963 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:19:51,383 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:19:51,634 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:19:51,692 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:19:51,696 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:19:51,705 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:19:51,708 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-web stdout | 2024-04-22 04:19:51,711 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:19:51,713 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:19:51,716 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:19:51,720 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:19:51,722 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:19:51,726 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:19:51,728 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:19:51,731 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:19:51,734 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:19:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:19:54.433683+00:00 (in 2.002162 seconds)
notificationworker stdout | 2024-04-22 04:19:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:02 UTC)" (scheduled at 2024-04-22 04:19:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:19:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:19:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 52, 431839), True, datetime.datetime(2024, 4, 22, 4, 19, 52, 431839), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:19:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:19:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:19:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:19:52,611 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:19:52,863 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:19:52,893 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:19:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:19:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:18.047111+00:00 (in 25.001053 seconds)
exportactionlogsworker stdout | 2024-04-22 04:19:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:53 UTC)" (scheduled at 2024-04-22 04:19:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:19:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:19:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 19, 53, 46475), True, datetime.datetime(2024, 4, 22, 4, 19, 53, 46475), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:19:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:19:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:19:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:19:53,774 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:19:53,805 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:19:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:19:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:02.430973+00:00 (in 7.996705 seconds)
notificationworker stdout | 2024-04-22 04:19:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:54 UTC)" (scheduled at 2024-04-22 04:19:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:19:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:19:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:20:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:19:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:19:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:12.806778+00:00 (in 14.999434 seconds)
quotatotalworker stdout | 2024-04-22 04:19:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:12 UTC)" (scheduled at 2024-04-22 04:19:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:19:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:19:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:19:57,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:19:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:19:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:28.130973+00:00 (in 29.999452 seconds)
buildlogsarchiver stdout | 2024-04-22 04:19:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:28 UTC)" (scheduled at 2024-04-22 04:19:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:19:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 19, 58, 131906), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:19:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:19:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:19:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:20:00,632 [282] [DEBUG] [app] Starting request: urn:request:e3308212-18b5-4773-b4e2-046ce6ff0a80 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:00,632 [286] [DEBUG] [app] Starting request: urn:request:e0378ea8-99de-46b4-ae68-b7281f5e6a3b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:20:00,635 [275] [DEBUG] [app] Starting request: urn:request:32d548c9-ef66-4a24-b674-b6fb8a5291a7 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:00,635 [283] [DEBUG] [app] Starting request: urn:request:fbfc4d35-36cb-4b06-b6d6-ef2045b64668 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:00,636 [275] [DEBUG] [app] Ending request: urn:request:32d548c9-ef66-4a24-b674-b6fb8a5291a7 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:32d548c9-ef66-4a24-b674-b6fb8a5291a7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:00,636 [283] [DEBUG] [app] Ending request: urn:request:fbfc4d35-36cb-4b06-b6d6-ef2045b64668 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:fbfc4d35-36cb-4b06-b6d6-ef2045b64668', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:20:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:20:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:00,639 [286] [DEBUG] [app] Starting request: urn:request:ca808add-04b7-458e-9fee-fd51f2d20b23 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:00,639 [286] [DEBUG] [app] Ending request: urn:request:ca808add-04b7-458e-9fee-fd51f2d20b23 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ca808add-04b7-458e-9fee-fd51f2d20b23', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:20:00,639 [285] [DEBUG] [app] Starting request: urn:request:65201c3b-599a-4cf9-8031-6ba7a3246e3b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:20:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:00,639 [285] [DEBUG] [app] Ending request: urn:request:65201c3b-599a-4cf9-8031-6ba7a3246e3b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:65201c3b-599a-4cf9-8031-6ba7a3246e3b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:20:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:00,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:00,660 [286] [DEBUG] [app] Ending request: urn:request:e0378ea8-99de-46b4-ae68-b7281f5e6a3b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e0378ea8-99de-46b4-ae68-b7281f5e6a3b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:00,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:20:00,660 [282] [DEBUG] [app] Ending request: urn:request:e3308212-18b5-4773-b4e2-046ce6ff0a80 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e3308212-18b5-4773-b4e2-046ce6ff0a80', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:00,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:20:00,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:20:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:20:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:20:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:31.842666+00:00 (in 29.999406 seconds)
gcworker stdout | 2024-04-22 04:20:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:31 UTC)" (scheduled at 2024-04-22 04:20:01.842666+00:00)
gcworker stdout | 2024-04-22 04:20:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550001843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:20:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:20:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:20:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:20:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:20:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:09.033206+00:00 (in 6.997044 seconds)
repositorygcworker stdout | 2024-04-22 04:20:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:02 UTC)" (scheduled at 2024-04-22 04:20:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:20:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:20:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:20:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:12.430973+00:00 (in 9.999437 seconds)
notificationworker stdout | 2024-04-22 04:20:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:12 UTC)" (scheduled at 2024-04-22 04:20:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:20:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:20:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 2, 431802), True, datetime.datetime(2024, 4, 22, 4, 20, 2, 431802), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:20:02,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:20:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:20:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:20:06,004 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:20:07,320 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:20:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:20:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:02.035613+00:00 (in 53.001940 seconds)
repositorygcworker stdout | 2024-04-22 04:20:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:09 UTC)" (scheduled at 2024-04-22 04:20:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:20:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:20:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 9, 34117), True, datetime.datetime(2024, 4, 22, 4, 20, 9, 34117), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:20:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:20:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:20:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:20:09,832 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:20:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:20:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:40.238173+00:00 (in 29.999488 seconds)
autopruneworker stdout | 2024-04-22 04:20:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:40 UTC)" (scheduled at 2024-04-22 04:20:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:20:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759550247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:20:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:20:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:47.245913+00:00 (in 36.996802 seconds)
namespacegcworker stdout | 2024-04-22 04:20:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:10 UTC)" (scheduled at 2024-04-22 04:20:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:20:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:20:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:20:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:20:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:20:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:20:10,878 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:20:11,224 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:20:11,427 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:20:11,816 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:20:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:22.430973+00:00 (in 9.999429 seconds)
notificationworker stdout | 2024-04-22 04:20:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:22 UTC)" (scheduled at 2024-04-22 04:20:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:20:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:20:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 12, 431858), True, datetime.datetime(2024, 4, 22, 4, 20, 12, 431858), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:20:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:20:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:20:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:20:12,492 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:20:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:20:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:27.806778+00:00 (in 14.999449 seconds)
quotatotalworker stdout | 2024-04-22 04:20:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:27 UTC)" (scheduled at 2024-04-22 04:20:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:20:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:20:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:20:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:27 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:20:12,831 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:20:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:20:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:17.065745+00:00 (in 2.998115 seconds)
securityworker stdout | 2024-04-22 04:20:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:44 UTC)" (scheduled at 2024-04-22 04:20:14.067050+00:00)
securityworker stdout | 2024-04-22 04:20:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:20:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:20:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:20:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:14,073 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:20:14,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:20:14,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:20:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:20:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:20:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 15, 14, 78302), 1, 2])
securityworker stdout | 2024-04-22 04:20:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 15, 14, 78302), 1, 2])
securityworker stdout | 2024-04-22 04:20:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:20:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:20:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:20:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:20:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:14,106 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:20:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:20:14,113 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:20:15,074 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
globalpromstats stdout | 2024-04-22 04:20:15,332 [68] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
globalpromstats stdout | 2024-04-22 04:20:15,333 [68] [INFO] [apscheduler.executors.default] Running job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2024-04-22 04:20:15 UTC)" (scheduled at 2024-04-22 04:20:15.332514+00:00)
globalpromstats stdout | 2024-04-22 04:20:15,333 [68] [DEBUG] [__main__] Attempting to report stats
globalpromstats stdout | 2024-04-22 04:20:15,333 [68] [DEBUG] [util.locking] Acquiring global lock GLOBAL_PROM_STATS
globalpromstats stdout | 2024-04-22 04:20:15,333 [68] [DEBUG] [redis_lock.acquire] Getting 'lock:GLOBAL_PROM_STATS' ...
globalpromstats stdout | 2024-04-22 04:20:15,333 [68] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 05:20:15.332514+00:00 (in 3599.998682 seconds)
globalpromstats stdout | 2024-04-22 04:20:15,336 [68] [INFO] [redis_lock.acquire] Got lock for 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2024-04-22 04:20:15,336 [68] [DEBUG] [util.locking] Acquired lock GLOBAL_PROM_STATS
globalpromstats stdout | 2024-04-22 04:20:15,336 [68] [DEBUG] [__main__] Reporting global stats
globalpromstats stdout | 2024-04-22 04:20:15,336 [68] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "repository" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2024-04-22 04:20:15,349 [68] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."id" NOT IN (SELECT "t2"."namespace_id" FROM "deletednamespace" AS "t2")))) AS "_wrapped"', [False, False])
globalpromstats stdout | 2024-04-22 04:20:15,354 [68] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "user" AS "t1" WHERE ((("t1"."organization" = %s) AND ("t1"."robot" = %s)) AND ("t1"."enabled" = %s))) AS "_wrapped"', [True, False, True])
globalpromstats stdout | 2024-04-22 04:20:15,357 [68] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT 1 FROM "robotaccounttoken" AS "t1") AS "_wrapped"', [])
globalpromstats stdout | 2024-04-22 04:20:15,361 [68] [DEBUG] [data.database] Disconnecting from database.
globalpromstats stdout | 2024-04-22 04:20:15,361 [68] [DEBUG] [util.locking] Releasing lock GLOBAL_PROM_STATS
globalpromstats stdout | 2024-04-22 04:20:15,361 [68] [DEBUG] [redis_lock.release] Releasing 'lock:GLOBAL_PROM_STATS'.
globalpromstats stdout | 2024-04-22 04:20:15,362 [68] [DEBUG] [util.locking] Released lock GLOBAL_PROM_STATS
globalpromstats stdout | 2024-04-22 04:20:15,362 [68] [INFO] [apscheduler.executors.default] Job "GlobalPrometheusStatsWorker._try_report_stats (trigger: interval[1:00:00], next run at: 2024-04-22 05:20:15 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:20:15,632 [285] [DEBUG] [app] Starting request: urn:request:5daed676-7c1c-4d49-b0bb-279035274d7e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:15,632 [282] [DEBUG] [app] Starting request: urn:request:8ebb71ed-ed43-45e3-b199-96ba62e5f5f8 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:15,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:20:15,635 [275] [DEBUG] [app] Starting request: urn:request:a6cb0781-1c9b-4009-a19b-da2f723e804e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:15,635 [279] [DEBUG] [app] Starting request: urn:request:3b398db5-836d-4d44-b2a8-3867f45b7a14 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:15,636 [275] [DEBUG] [app] Ending request: urn:request:a6cb0781-1c9b-4009-a19b-da2f723e804e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a6cb0781-1c9b-4009-a19b-da2f723e804e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:15,636 [279] [DEBUG] [app] Ending request: urn:request:3b398db5-836d-4d44-b2a8-3867f45b7a14 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3b398db5-836d-4d44-b2a8-3867f45b7a14', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:20:15,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-registry stdout | 2024-04-22 04:20:15,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:20:15,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:15,639 [286] [DEBUG] [app] Starting request: urn:request:24acf8bf-1bdc-42f3-84ba-73f6beb632ed (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:15,639 [284] [DEBUG] [app] Starting request: urn:request:a80bfdb8-988e-4c08-b8d5-0a2b06e3dd5a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:15,639 [286] [DEBUG] [app] Ending request: urn:request:24acf8bf-1bdc-42f3-84ba-73f6beb632ed (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:24acf8bf-1bdc-42f3-84ba-73f6beb632ed', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:20:15,639 [284] [DEBUG] [app] Ending request: urn:request:a80bfdb8-988e-4c08-b8d5-0a2b06e3dd5a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a80bfdb8-988e-4c08-b8d5-0a2b06e3dd5a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:20:15,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:15,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:20:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:15,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:15,660 [285] [DEBUG] [app] Ending request: urn:request:5daed676-7c1c-4d49-b0bb-279035274d7e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:5daed676-7c1c-4d49-b0bb-279035274d7e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:15,661 [282] [DEBUG] [app] Ending request: urn:request:8ebb71ed-ed43-45e3-b199-96ba62e5f5f8 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8ebb71ed-ed43-45e3-b199-96ba62e5f5f8', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:15,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:20:15,661 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:20:15,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:20:15,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:20:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:20:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:44.067050+00:00 (in 27.000726 seconds)
securityworker stdout | 2024-04-22 04:20:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:47 UTC)" (scheduled at 2024-04-22 04:20:17.065745+00:00)
securityworker stdout | 2024-04-22 04:20:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:20:17,067 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:20:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:20:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:20:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:20:17,087 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:20:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:17,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 15, 17, 70889), 1, 2])
securityworker stdout | 2024-04-22 04:20:17,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:17,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 15, 17, 70889), 1, 2])
securityworker stdout | 2024-04-22 04:20:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:20:17,101 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:20:17,613 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:20:17,743 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:20:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:20:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:53.045554+00:00 (in 34.997936 seconds)
exportactionlogsworker stdout | 2024-04-22 04:20:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:18 UTC)" (scheduled at 2024-04-22 04:20:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:20:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:20:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:20:18,318 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:20:18,893 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:20:19,783 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:20:20,079 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:20:20,082 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:20:20,084 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:20:20,996 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:20:21,392 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:20:21,643 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:20:21,703 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:20:21,706 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:20:21,715 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:20:21,718 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-web stdout | 2024-04-22 04:20:21,720 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:20:21,723 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:20:21,727 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:20:21,730 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:20:21,733 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:20:21,736 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:20:21,739 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:20:21,742 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:20:21,745 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:20:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:32.430973+00:00 (in 9.999461 seconds)
notificationworker stdout | 2024-04-22 04:20:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:32 UTC)" (scheduled at 2024-04-22 04:20:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:20:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:20:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 22, 431798), True, datetime.datetime(2024, 4, 22, 4, 20, 22, 431798), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:20:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:20:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:20:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:20:22,648 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:20:22,876 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:20:22,930 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
quotatotalworker stdout | 2024-04-22 04:20:23,811 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:20:23,814 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:20:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:20:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:24.738365+00:00 (in 59.999472 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:20:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:24 UTC)" (scheduled at 2024-04-22 04:20:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:20:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:20:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:20:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:20:27,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:20:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:42.806778+00:00 (in 14.999515 seconds)
quotatotalworker stdout | 2024-04-22 04:20:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:42 UTC)" (scheduled at 2024-04-22 04:20:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:20:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:20:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:20:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:20:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:20:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:58.130973+00:00 (in 29.999454 seconds)
buildlogsarchiver stdout | 2024-04-22 04:20:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:58 UTC)" (scheduled at 2024-04-22 04:20:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:20:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 20, 28, 131902), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:20:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:20:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:20:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:20:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:31.933721+00:00 (in 2.002280 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:29 UTC)" (scheduled at 2024-04-22 04:20:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 29, 931727), True, datetime.datetime(2024, 4, 22, 4, 20, 29, 931727), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,957 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,957 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:20:29,957 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:20:30,632 [282] [DEBUG] [app] Starting request: urn:request:1bc3b22d-2dcd-4b79-a1e0-80958ab8b359 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:30,632 [284] [DEBUG] [app] Starting request: urn:request:ba579932-04ab-41bb-a94c-fa9129ef225d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:30,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:20:30,636 [281] [DEBUG] [app] Starting request: urn:request:ec7885b3-7cdd-4952-bd39-939be1d5de0e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:30,636 [283] [DEBUG] [app] Starting request: urn:request:8fd194fd-a264-4680-915f-c3b4b623e4c9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:30,636 [281] [DEBUG] [app] Ending request: urn:request:ec7885b3-7cdd-4952-bd39-939be1d5de0e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ec7885b3-7cdd-4952-bd39-939be1d5de0e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:30,636 [283] [DEBUG] [app] Ending request: urn:request:8fd194fd-a264-4680-915f-c3b4b623e4c9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8fd194fd-a264-4680-915f-c3b4b623e4c9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:30,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:20:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:30,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:30,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:30,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:30,639 [285] [DEBUG] [app] Starting request: urn:request:5ec6e9c1-529a-4bea-95b4-fb29d3f98a83 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:30,639 [286] [DEBUG] [app] Starting request: urn:request:5e00e046-7ce8-47d7-930d-25d71a0339bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:30,639 [285] [DEBUG] [app] Ending request: urn:request:5ec6e9c1-529a-4bea-95b4-fb29d3f98a83 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5ec6e9c1-529a-4bea-95b4-fb29d3f98a83', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:20:30,640 [286] [DEBUG] [app] Ending request: urn:request:5e00e046-7ce8-47d7-930d-25d71a0339bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5e00e046-7ce8-47d7-930d-25d71a0339bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:20:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:30,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:30,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:20:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:30,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:30,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:30,661 [284] [DEBUG] [app] Ending request: urn:request:ba579932-04ab-41bb-a94c-fa9129ef225d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ba579932-04ab-41bb-a94c-fa9129ef225d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:30,661 [282] [DEBUG] [app] Ending request: urn:request:1bc3b22d-2dcd-4b79-a1e0-80958ab8b359 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1bc3b22d-2dcd-4b79-a1e0-80958ab8b359', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:20:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.029)
gunicorn-web stdout | 2024-04-22 04:20:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:20:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:20:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:20:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:01.842666+00:00 (in 29.999453 seconds)
gcworker stdout | 2024-04-22 04:20:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:01 UTC)" (scheduled at 2024-04-22 04:20:31.842666+00:00)
gcworker stdout | 2024-04-22 04:20:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550031843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:20:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:20:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:20:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:20:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:20:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:29.931002+00:00 (in 57.996805 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:20:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:31 UTC)" (scheduled at 2024-04-22 04:20:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:20:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:20:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:20:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:42.430973+00:00 (in 9.999554 seconds)
notificationworker stdout | 2024-04-22 04:20:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:42 UTC)" (scheduled at 2024-04-22 04:20:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:20:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:20:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 32, 431684), True, datetime.datetime(2024, 4, 22, 4, 20, 32, 431684), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:20:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:20:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:20:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:20:36,040 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:20:37,334 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:20:39,869 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:20:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:20:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:10.238173+00:00 (in 29.999465 seconds)
autopruneworker stdout | 2024-04-22 04:20:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:10 UTC)" (scheduled at 2024-04-22 04:20:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:20:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759580247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:20:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:20:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:20:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:20:40,907 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:20:41,235 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:20:41,463 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:20:41,826 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:20:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:52.430973+00:00 (in 9.999518 seconds)
notificationworker stdout | 2024-04-22 04:20:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:52 UTC)" (scheduled at 2024-04-22 04:20:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:20:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:20:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 42, 431735), True, datetime.datetime(2024, 4, 22, 4, 20, 42, 431735), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:20:42,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:20:42,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:20:42,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:20:52 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:20:42,520 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:20:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:20:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:57.806778+00:00 (in 14.999472 seconds)
quotatotalworker stdout | 2024-04-22 04:20:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:57 UTC)" (scheduled at 2024-04-22 04:20:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:20:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:20:42,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:20:42,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:20:57 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:20:42,856 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:20:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:20:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:47.065745+00:00 (in 2.998168 seconds)
securityworker stdout | 2024-04-22 04:20:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:14 UTC)" (scheduled at 2024-04-22 04:20:44.067050+00:00)
securityworker stdout | 2024-04-22 04:20:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:20:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:20:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:20:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:44,073 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:20:44,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:20:44,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:20:44,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:20:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:20:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:44,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 15, 44, 78809), 1, 2])
securityworker stdout | 2024-04-22 04:20:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 15, 44, 78809), 1, 2])
securityworker stdout | 2024-04-22 04:20:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:20:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:20:44,106 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:44,106 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:20:44,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:20:44,107 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:20:44,107 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:20:44,108 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:20:44,108 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:20:44,108 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:14 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:20:44,138 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:20:45,110 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:20:45,632 [286] [DEBUG] [app] Starting request: urn:request:f1c1f774-e9cd-4c45-a758-0e272de4bf78 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:45,632 [285] [DEBUG] [app] Starting request: urn:request:a26be022-b029-4ef9-8f7b-95f686e19861 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:20:45,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:20:45,635 [280] [DEBUG] [app] Starting request: urn:request:a1455ab8-0d5d-4f27-a990-c1f561b3293a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:45,635 [275] [DEBUG] [app] Starting request: urn:request:97939889-eb6c-4de7-843e-21d9e85849cb (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:20:45,636 [280] [DEBUG] [app] Ending request: urn:request:a1455ab8-0d5d-4f27-a990-c1f561b3293a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a1455ab8-0d5d-4f27-a990-c1f561b3293a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:45,636 [275] [DEBUG] [app] Ending request: urn:request:97939889-eb6c-4de7-843e-21d9e85849cb (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:97939889-eb6c-4de7-843e-21d9e85849cb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:20:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:20:45,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:20:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:45,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:20:45,638 [286] [DEBUG] [app] Starting request: urn:request:0b04fb8f-6921-4f09-bb03-6155ae376589 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:45,639 [286] [DEBUG] [app] Ending request: urn:request:0b04fb8f-6921-4f09-bb03-6155ae376589 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:0b04fb8f-6921-4f09-bb03-6155ae376589', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:20:45,639 [282] [DEBUG] [app] Starting request: urn:request:abe77fbf-3122-41c7-a144-a70a2894a1d2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:20:45,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:20:45,639 [282] [DEBUG] [app] Ending request: urn:request:abe77fbf-3122-41c7-a144-a70a2894a1d2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abe77fbf-3122-41c7-a144-a70a2894a1d2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:20:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:20:45,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:20:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:20:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:20:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:20:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:20:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:45,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:20:45,659 [286] [DEBUG] [app] Ending request: urn:request:f1c1f774-e9cd-4c45-a758-0e272de4bf78 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f1c1f774-e9cd-4c45-a758-0e272de4bf78', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:45,659 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:20:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:20:45,665 [285] [DEBUG] [app] Ending request: urn:request:a26be022-b029-4ef9-8f7b-95f686e19861 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a26be022-b029-4ef9-8f7b-95f686e19861', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:20:45,665 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:20:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.034 118 0.034)
gunicorn-web stdout | 2024-04-22 04:20:45,665 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:20:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:20:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:20:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:14.067050+00:00 (in 27.000813 seconds)
securityworker stdout | 2024-04-22 04:20:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:17 UTC)" (scheduled at 2024-04-22 04:20:47.065745+00:00)
securityworker stdout | 2024-04-22 04:20:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:20:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:20:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:20:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:20:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:20:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:20:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:20:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:10.248668+00:00 (in 23.002245 seconds)
namespacegcworker stdout | 2024-04-22 04:20:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:47 UTC)" (scheduled at 2024-04-22 04:20:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:20:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:20:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 47, 246872), True, datetime.datetime(2024, 4, 22, 4, 20, 47, 246872), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:20:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:20:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:20:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:20:47,619 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:20:47,755 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:20:48,336 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:20:48,928 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:20:49,815 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:20:50,087 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:20:50,091 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:20:50,094 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:20:51,022 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:20:51,404 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:20:51,651 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:20:51,712 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:20:51,717 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:20:51,724 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:20:51,727 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-web stdout | 2024-04-22 04:20:51,730 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:20:51,732 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:20:51,735 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:20:51,739 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:20:51,743 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:20:51,746 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:20:51,749 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:20:51,751 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:20:51,754 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:20:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:20:54.433683+00:00 (in 2.002241 seconds)
notificationworker stdout | 2024-04-22 04:20:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:02 UTC)" (scheduled at 2024-04-22 04:20:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:20:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:20:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 52, 431714), True, datetime.datetime(2024, 4, 22, 4, 20, 52, 431714), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:20:52,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:20:52,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:20:52,445 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:20:52,685 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:20:52,913 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:20:52,940 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:20:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:20:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:18.047111+00:00 (in 25.001094 seconds)
exportactionlogsworker stdout | 2024-04-22 04:20:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:53 UTC)" (scheduled at 2024-04-22 04:20:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:20:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:20:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 20, 53, 46443), True, datetime.datetime(2024, 4, 22, 4, 20, 53, 46443), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:20:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:20:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:20:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:20:53,847 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:20:53,850 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:20:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:20:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:02.430973+00:00 (in 7.996851 seconds)
notificationworker stdout | 2024-04-22 04:20:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:54 UTC)" (scheduled at 2024-04-22 04:20:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:20:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:20:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:21:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:20:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:20:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:12.806778+00:00 (in 14.999461 seconds)
quotatotalworker stdout | 2024-04-22 04:20:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:12 UTC)" (scheduled at 2024-04-22 04:20:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:20:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:20:57,833 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:20:57,833 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:20:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:20:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:28.130973+00:00 (in 29.999483 seconds)
buildlogsarchiver stdout | 2024-04-22 04:20:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:28 UTC)" (scheduled at 2024-04-22 04:20:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:20:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 20, 58, 131871), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:20:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:20:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:20:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:21:00,632 [285] [DEBUG] [app] Starting request: urn:request:3110e4e1-87fa-4924-9cb4-9f3963b1fc25 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:00,632 [286] [DEBUG] [app] Starting request: urn:request:306fd289-2259-4cc9-bb24-860513568712 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:00,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:00,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:21:00,635 [281] [DEBUG] [app] Starting request: urn:request:8df9163f-f4ee-4baf-8bec-48046a4b705d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:00,635 [275] [DEBUG] [app] Starting request: urn:request:b753fd90-7010-40de-9e83-8ecca9e9958f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:00,635 [281] [DEBUG] [app] Ending request: urn:request:8df9163f-f4ee-4baf-8bec-48046a4b705d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:8df9163f-f4ee-4baf-8bec-48046a4b705d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:00,636 [275] [DEBUG] [app] Ending request: urn:request:b753fd90-7010-40de-9e83-8ecca9e9958f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b753fd90-7010-40de-9e83-8ecca9e9958f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:21:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:21:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:00,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:00,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:00,639 [282] [DEBUG] [app] Starting request: urn:request:68c0270d-4dfb-4b6d-ae96-591f2c444cdb (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:00,639 [285] [DEBUG] [app] Starting request: urn:request:611c71cd-6457-4638-945d-ac9e3cdb7ab6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:00,639 [282] [DEBUG] [app] Ending request: urn:request:68c0270d-4dfb-4b6d-ae96-591f2c444cdb (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:68c0270d-4dfb-4b6d-ae96-591f2c444cdb', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:00,639 [285] [DEBUG] [app] Ending request: urn:request:611c71cd-6457-4638-945d-ac9e3cdb7ab6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:611c71cd-6457-4638-945d-ac9e3cdb7ab6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:00,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:21:00,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:00,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:00,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:00,661 [285] [DEBUG] [app] Ending request: urn:request:3110e4e1-87fa-4924-9cb4-9f3963b1fc25 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:3110e4e1-87fa-4924-9cb4-9f3963b1fc25', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:21:00,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:21:00,662 [286] [DEBUG] [app] Ending request: urn:request:306fd289-2259-4cc9-bb24-860513568712 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:306fd289-2259-4cc9-bb24-860513568712', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:21:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 04:21:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:21:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:21:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:31.842666+00:00 (in 29.999453 seconds)
gcworker stdout | 2024-04-22 04:21:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:31 UTC)" (scheduled at 2024-04-22 04:21:01.842666+00:00)
gcworker stdout | 2024-04-22 04:21:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550061843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:21:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:21:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:21:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:21:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:21:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:09.033206+00:00 (in 6.997051 seconds)
repositorygcworker stdout | 2024-04-22 04:21:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:02 UTC)" (scheduled at 2024-04-22 04:21:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:21:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:21:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:21:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:12.430973+00:00 (in 9.999476 seconds)
notificationworker stdout | 2024-04-22 04:21:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:12 UTC)" (scheduled at 2024-04-22 04:21:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:21:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:21:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 2, 431765), True, datetime.datetime(2024, 4, 22, 4, 21, 2, 431765), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:21:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:21:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:21:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:21:06,077 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:21:07,341 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:21:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:21:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:02.035613+00:00 (in 53.001904 seconds)
repositorygcworker stdout | 2024-04-22 04:21:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:09 UTC)" (scheduled at 2024-04-22 04:21:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:21:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:21:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 9, 34180), True, datetime.datetime(2024, 4, 22, 4, 21, 9, 34180), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:21:09,052 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:21:09,052 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:21:09,052 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:21:09,905 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:21:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:21:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:40.238173+00:00 (in 29.999490 seconds)
autopruneworker stdout | 2024-04-22 04:21:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:40 UTC)" (scheduled at 2024-04-22 04:21:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:21:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759610247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:21:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:21:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:30.247852+00:00 (in 19.998728 seconds)
namespacegcworker stdout | 2024-04-22 04:21:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:10 UTC)" (scheduled at 2024-04-22 04:21:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:21:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:21:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:21:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:21:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:21:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:21:10,940 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:21:11,244 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:21:11,500 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:21:11,863 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:21:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:22.430973+00:00 (in 9.999470 seconds)
notificationworker stdout | 2024-04-22 04:21:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:22 UTC)" (scheduled at 2024-04-22 04:21:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:21:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:21:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 12, 431789), True, datetime.datetime(2024, 4, 22, 4, 21, 12, 431789), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:21:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:21:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:21:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:21:12,538 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:21:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:21:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:27.806778+00:00 (in 14.999464 seconds)
quotatotalworker stdout | 2024-04-22 04:21:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:27 UTC)" (scheduled at 2024-04-22 04:21:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:21:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:21:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:21:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:27 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:21:12,893 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:21:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:21:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:17.065745+00:00 (in 2.998149 seconds)
securityworker stdout | 2024-04-22 04:21:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:44 UTC)" (scheduled at 2024-04-22 04:21:14.067050+00:00)
securityworker stdout | 2024-04-22 04:21:14,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:21:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:21:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:21:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:21:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:21:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:21:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:21:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:21:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 16, 14, 77836), 1, 2])
securityworker stdout | 2024-04-22 04:21:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 16, 14, 77836), 1, 2])
securityworker stdout | 2024-04-22 04:21:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:21:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:21:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:21:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:21:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:21:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:21:14,151 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:21:15,134 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:21:15,633 [284] [DEBUG] [app] Starting request: urn:request:a30223ee-6427-4ed9-aa18-669f4759c617 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:15,633 [282] [DEBUG] [app] Starting request: urn:request:c160e152-cfab-4ce0-9a62-09953d7f4dd6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:15,635 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:15,635 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:21:15,637 [281] [DEBUG] [app] Starting request: urn:request:77c84904-808a-4ec2-8f7d-81caa06b38bd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:15,637 [280] [DEBUG] [app] Starting request: urn:request:331cf72b-7686-4032-b239-8084292d510e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:15,637 [280] [DEBUG] [app] Ending request: urn:request:331cf72b-7686-4032-b239-8084292d510e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:331cf72b-7686-4032-b239-8084292d510e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:15,637 [281] [DEBUG] [app] Ending request: urn:request:77c84904-808a-4ec2-8f7d-81caa06b38bd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:77c84904-808a-4ec2-8f7d-81caa06b38bd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:15,637 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:21:15,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:15,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:15,638 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:21:15,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:15,639 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:15,641 [285] [DEBUG] [app] Starting request: urn:request:7f567cbd-361a-421f-81b0-ce1235ab6576 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:15,641 [286] [DEBUG] [app] Starting request: urn:request:76b5bc7c-4190-4573-93d4-9fbf09bab69a (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:15,641 [285] [DEBUG] [app] Ending request: urn:request:7f567cbd-361a-421f-81b0-ce1235ab6576 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f567cbd-361a-421f-81b0-ce1235ab6576', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:15,641 [286] [DEBUG] [app] Ending request: urn:request:76b5bc7c-4190-4573-93d4-9fbf09bab69a (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:76b5bc7c-4190-4573-93d4-9fbf09bab69a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:21:15,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:15,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:15,642 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:15,642 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:15,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:15,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:15,651 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:15,652 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:15,662 [282] [DEBUG] [app] Ending request: urn:request:c160e152-cfab-4ce0-9a62-09953d7f4dd6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c160e152-cfab-4ce0-9a62-09953d7f4dd6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:15,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:21:15,663 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.031)
gunicorn-web stdout | 2024-04-22 04:21:15,663 [284] [DEBUG] [app] Ending request: urn:request:a30223ee-6427-4ed9-aa18-669f4759c617 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a30223ee-6427-4ed9-aa18-669f4759c617', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:15,663 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 04:21:15,663 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:21:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:21:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:44.067050+00:00 (in 27.000739 seconds)
securityworker stdout | 2024-04-22 04:21:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:47 UTC)" (scheduled at 2024-04-22 04:21:17.065745+00:00)
securityworker stdout | 2024-04-22 04:21:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:21:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:21:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:21:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:21:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:21:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:21:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 16, 17, 70568), 1, 2])
securityworker stdout | 2024-04-22 04:21:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 16, 17, 70568), 1, 2])
securityworker stdout | 2024-04-22 04:21:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:21:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:21:17,656 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:21:17,768 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:21:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:21:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:53.045554+00:00 (in 34.997802 seconds)
exportactionlogsworker stdout | 2024-04-22 04:21:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:18 UTC)" (scheduled at 2024-04-22 04:21:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:21:18,048 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:21:18,048 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:21:18,372 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:21:18,955 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:21:19,830 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:21:20,095 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:21:20,100 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:21:20,103 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:21:21,053 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:21:21,440 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:21:21,659 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:21:21,721 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:21:21,724 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:21:21,733 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:21:21,737 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-web stdout | 2024-04-22 04:21:21,739 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:21:21,742 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:21:21,744 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-registry stdout | 2024-04-22 04:21:21,747 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-web stdout | 2024-04-22 04:21:21,753 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-registry stdout | 2024-04-22 04:21:21,757 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:21:21,759 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:21:21,762 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:21:21,764 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:21:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:32.430973+00:00 (in 9.999497 seconds)
notificationworker stdout | 2024-04-22 04:21:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:32 UTC)" (scheduled at 2024-04-22 04:21:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:21:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:21:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 22, 431719), True, datetime.datetime(2024, 4, 22, 4, 21, 22, 431719), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:21:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:21:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:21:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:21:22,722 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:21:22,949 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:21:22,957 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
quotatotalworker stdout | 2024-04-22 04:21:23,869 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:21:23,884 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:21:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:21:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:24.738365+00:00 (in 59.999458 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:21:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:24 UTC)" (scheduled at 2024-04-22 04:21:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:21:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:21:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:21:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:21:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:21:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:42.806778+00:00 (in 14.999459 seconds)
quotatotalworker stdout | 2024-04-22 04:21:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:42 UTC)" (scheduled at 2024-04-22 04:21:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:21:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:21:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:21:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:21:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:21:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:58.130973+00:00 (in 29.999479 seconds)
buildlogsarchiver stdout | 2024-04-22 04:21:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:58 UTC)" (scheduled at 2024-04-22 04:21:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:21:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 21, 28, 131845), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:21:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:21:28,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:21:28,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:21:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:31.933721+00:00 (in 2.002214 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:29 UTC)" (scheduled at 2024-04-22 04:21:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 29, 931797), True, datetime.datetime(2024, 4, 22, 4, 21, 29, 931797), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,947 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:21:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:21:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:21:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:47.245913+00:00 (in 16.997626 seconds)
namespacegcworker stdout | 2024-04-22 04:21:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:26:30 UTC)" (scheduled at 2024-04-22 04:21:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 04:21:30,248 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 21, 30, 248476), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:21:30,262 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 21, 30, 248476), True, datetime.datetime(2024, 4, 22, 4, 21, 30, 248476), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:21:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 21, 30, 248476), True, datetime.datetime(2024, 4, 22, 4, 21, 30, 248476), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 4, 21, 30, 248476), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:21:30,270 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:21:30,270 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:26:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:21:30,631 [285] [DEBUG] [app] Starting request: urn:request:fdb3abf5-aa0d-42ed-8632-672e349b3c83 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:30,631 [284] [DEBUG] [app] Starting request: urn:request:cb006558-db04-4ecd-ab57-e981180d436a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:30,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:21:30,634 [281] [DEBUG] [app] Starting request: urn:request:a2e0305c-aa32-4b13-a324-271e7db0b212 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:30,634 [283] [DEBUG] [app] Starting request: urn:request:ccfcfd1b-f0bd-4d2e-9b03-487a523d988e (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:30,635 [281] [DEBUG] [app] Ending request: urn:request:a2e0305c-aa32-4b13-a324-271e7db0b212 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a2e0305c-aa32-4b13-a324-271e7db0b212', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:30,635 [283] [DEBUG] [app] Ending request: urn:request:ccfcfd1b-f0bd-4d2e-9b03-487a523d988e (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ccfcfd1b-f0bd-4d2e-9b03-487a523d988e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:30,635 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:21:30,635 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:30,635 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:30,635 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:21:30,636 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:30,638 [284] [DEBUG] [app] Starting request: urn:request:f7922a9f-0173-4aa6-bfc6-112d7a4a92e2 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:30,638 [284] [DEBUG] [app] Ending request: urn:request:f7922a9f-0173-4aa6-bfc6-112d7a4a92e2 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f7922a9f-0173-4aa6-bfc6-112d7a4a92e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:30,638 [285] [DEBUG] [app] Starting request: urn:request:35cdd684-ffbb-438a-b760-8b4242ce32f7 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:30,638 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:21:30,638 [285] [DEBUG] [app] Ending request: urn:request:35cdd684-ffbb-438a-b760-8b4242ce32f7 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:35cdd684-ffbb-438a-b760-8b4242ce32f7', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:30,638 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:21:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:30,639 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:30,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:30,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:30,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:30,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:30,659 [284] [DEBUG] [app] Ending request: urn:request:cb006558-db04-4ecd-ab57-e981180d436a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:cb006558-db04-4ecd-ab57-e981180d436a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:30,659 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:21:30,659 [285] [DEBUG] [app] Ending request: urn:request:fdb3abf5-aa0d-42ed-8632-672e349b3c83 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fdb3abf5-aa0d-42ed-8632-672e349b3c83', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:30,659 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:21:30,660 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:21:30,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:21:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:21:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:01.842666+00:00 (in 29.999477 seconds)
gcworker stdout | 2024-04-22 04:21:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:01 UTC)" (scheduled at 2024-04-22 04:21:31.842666+00:00)
gcworker stdout | 2024-04-22 04:21:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550091843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:21:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:21:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:21:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:21:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:21:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:29.931002+00:00 (in 57.996768 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:21:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:31 UTC)" (scheduled at 2024-04-22 04:21:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:21:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:21:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:21:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:42.430973+00:00 (in 9.999536 seconds)
notificationworker stdout | 2024-04-22 04:21:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:42 UTC)" (scheduled at 2024-04-22 04:21:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:21:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:21:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 32, 431804), True, datetime.datetime(2024, 4, 22, 4, 21, 32, 431804), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:21:32,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:21:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:21:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:21:36,109 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:21:37,367 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:21:39,941 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:21:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:21:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:10.238173+00:00 (in 29.999494 seconds)
autopruneworker stdout | 2024-04-22 04:21:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:10 UTC)" (scheduled at 2024-04-22 04:21:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:21:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759640247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:21:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:21:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:21:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:21:40,962 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:21:41,266 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:21:41,517 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:21:41,897 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:21:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:52.430973+00:00 (in 9.999498 seconds)
notificationworker stdout | 2024-04-22 04:21:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:52 UTC)" (scheduled at 2024-04-22 04:21:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:21:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:21:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 42, 431884), True, datetime.datetime(2024, 4, 22, 4, 21, 42, 431884), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:21:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:21:42,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:21:42,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:21:52 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:21:42,575 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:21:42,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:21:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:57.806778+00:00 (in 14.999536 seconds)
quotatotalworker stdout | 2024-04-22 04:21:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:57 UTC)" (scheduled at 2024-04-22 04:21:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:21:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:21:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:21:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:21:57 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:21:42,929 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:21:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:21:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:47.065745+00:00 (in 2.998144 seconds)
securityworker stdout | 2024-04-22 04:21:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:14 UTC)" (scheduled at 2024-04-22 04:21:44.067050+00:00)
securityworker stdout | 2024-04-22 04:21:44,068 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:21:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:21:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:21:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:21:44,073 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:21:44,077 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:21:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:21:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:21:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:44,107 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 16, 44, 78397), 1, 2])
securityworker stdout | 2024-04-22 04:21:44,110 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:44,110 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:44,111 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 16, 44, 78397), 1, 2])
securityworker stdout | 2024-04-22 04:21:44,115 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:21:44,115 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:21:44,115 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:44,115 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:21:44,115 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:21:44,115 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:21:44,116 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:21:44,117 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:21:44,117 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:21:44,117 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:14 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:21:44,188 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:21:45,170 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:21:45,632 [286] [DEBUG] [app] Starting request: urn:request:1d7ba89d-2373-4b74-8e90-1892f585f7e3 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:45,632 [285] [DEBUG] [app] Starting request: urn:request:eb77d68e-bca2-463f-ae71-f56aa71a94bd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:21:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:21:45,635 [275] [DEBUG] [app] Starting request: urn:request:9088bfc8-448d-4349-8826-83c850a1ba56 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:45,635 [283] [DEBUG] [app] Starting request: urn:request:cf93d8a6-4402-45fb-8405-30cb02097c01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:21:45,636 [275] [DEBUG] [app] Ending request: urn:request:9088bfc8-448d-4349-8826-83c850a1ba56 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9088bfc8-448d-4349-8826-83c850a1ba56', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:21:45,636 [283] [DEBUG] [app] Ending request: urn:request:cf93d8a6-4402-45fb-8405-30cb02097c01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cf93d8a6-4402-45fb-8405-30cb02097c01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:21:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:21:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:21:45,638 [285] [DEBUG] [app] Starting request: urn:request:7f40a44f-6287-40fa-a10c-b630410b07ee (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:21:45,639 [285] [DEBUG] [app] Ending request: urn:request:7f40a44f-6287-40fa-a10c-b630410b07ee (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:7f40a44f-6287-40fa-a10c-b630410b07ee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:45,639 [284] [DEBUG] [app] Starting request: urn:request:d319c203-5bf3-464b-95f4-4a5db7dfa4e3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:21:45,639 [284] [DEBUG] [app] Ending request: urn:request:d319c203-5bf3-464b-95f4-4a5db7dfa4e3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d319c203-5bf3-464b-95f4-4a5db7dfa4e3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:21:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:21:45,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:21:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:21:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:21:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:21:45,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:21:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:45,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:21:45,659 [285] [DEBUG] [app] Ending request: urn:request:eb77d68e-bca2-463f-ae71-f56aa71a94bd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:eb77d68e-bca2-463f-ae71-f56aa71a94bd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:45,659 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:21:45,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:21:45,660 [286] [DEBUG] [app] Ending request: urn:request:1d7ba89d-2373-4b74-8e90-1892f585f7e3 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:1d7ba89d-2373-4b74-8e90-1892f585f7e3', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:21:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:21:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.030)
gunicorn-web stdout | 2024-04-22 04:21:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:21:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:21:47,066 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:21:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:14.067050+00:00 (in 27.000730 seconds)
securityworker stdout | 2024-04-22 04:21:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:17 UTC)" (scheduled at 2024-04-22 04:21:47.065745+00:00)
securityworker stdout | 2024-04-22 04:21:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:21:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:21:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:21:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:21:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:21:47,083 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:21:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:21:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:10.248668+00:00 (in 23.002253 seconds)
namespacegcworker stdout | 2024-04-22 04:21:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:47 UTC)" (scheduled at 2024-04-22 04:21:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:21:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:21:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 47, 246687), True, datetime.datetime(2024, 4, 22, 4, 21, 47, 246687), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:21:47,262 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:21:47,262 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:21:47,262 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:21:47,685 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:21:47,788 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:21:48,409 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:21:48,967 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:21:49,866 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:21:50,104 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:21:50,108 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:21:50,111 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:21:51,075 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:21:51,476 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:21:51,672 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:21:51,729 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:21:51,732 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:21:51,742 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-web stdout | 2024-04-22 04:21:51,746 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:21:51,748 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:21:51,751 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:21:51,754 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:21:51,756 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:21:51,761 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:21:51,767 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-registry stdout | 2024-04-22 04:21:51,770 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:21:51,772 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:21:51,775 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:21:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:21:54.433683+00:00 (in 2.002208 seconds)
notificationworker stdout | 2024-04-22 04:21:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:02 UTC)" (scheduled at 2024-04-22 04:21:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:21:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:21:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 52, 431777), True, datetime.datetime(2024, 4, 22, 4, 21, 52, 431777), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:21:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:21:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:21:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:21:52,759 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:21:52,967 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:21:52,990 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:21:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:21:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:18.047111+00:00 (in 25.001095 seconds)
exportactionlogsworker stdout | 2024-04-22 04:21:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:53 UTC)" (scheduled at 2024-04-22 04:21:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:21:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:21:53,046 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 21, 53, 46279), True, datetime.datetime(2024, 4, 22, 4, 21, 53, 46279), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:21:53,072 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:21:53,072 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:21:53,072 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:53 UTC)" executed successfully
repositoryactioncounter stdout | 2024-04-22 04:21:53,901 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:21:53,904 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:21:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:21:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:02.430973+00:00 (in 7.996777 seconds)
notificationworker stdout | 2024-04-22 04:21:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:54 UTC)" (scheduled at 2024-04-22 04:21:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:21:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:21:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:22:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:21:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:21:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:12.806778+00:00 (in 14.999494 seconds)
quotatotalworker stdout | 2024-04-22 04:21:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:12 UTC)" (scheduled at 2024-04-22 04:21:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:21:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:21:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:21:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:21:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:21:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:28.130973+00:00 (in 29.999525 seconds)
buildlogsarchiver stdout | 2024-04-22 04:21:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:28 UTC)" (scheduled at 2024-04-22 04:21:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:21:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 21, 58, 131797), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:21:58,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:21:58,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:21:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:22:00,632 [282] [DEBUG] [app] Starting request: urn:request:fc35e082-f483-4ac8-882e-eb71a850e2c4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:00,632 [284] [DEBUG] [app] Starting request: urn:request:f441c376-31f0-4fc7-a71d-04ed48aa53b9 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:22:00,636 [275] [DEBUG] [app] Starting request: urn:request:484b5813-da1c-43f8-98a9-81c22a3bde88 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:00,636 [283] [DEBUG] [app] Starting request: urn:request:f4aa866f-2466-43f4-827f-f706ef971c01 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:00,637 [275] [DEBUG] [app] Ending request: urn:request:484b5813-da1c-43f8-98a9-81c22a3bde88 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:484b5813-da1c-43f8-98a9-81c22a3bde88', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:22:00,637 [283] [DEBUG] [app] Ending request: urn:request:f4aa866f-2466-43f4-827f-f706ef971c01 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:f4aa866f-2466-43f4-827f-f706ef971c01', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:22:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:22:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:00,639 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:00,639 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:00,640 [286] [DEBUG] [app] Starting request: urn:request:f3735015-8332-4f8b-a272-99122aa2f7c4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:00,641 [286] [DEBUG] [app] Ending request: urn:request:f3735015-8332-4f8b-a272-99122aa2f7c4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:f3735015-8332-4f8b-a272-99122aa2f7c4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:00,641 [285] [DEBUG] [app] Starting request: urn:request:ff52fe09-75a3-49de-83ac-e2ca7c219cd6 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:00,641 [285] [DEBUG] [app] Ending request: urn:request:ff52fe09-75a3-49de-83ac-e2ca7c219cd6 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:ff52fe09-75a3-49de-83ac-e2ca7c219cd6', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:00,641 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:22:00,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:00,642 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:00,642 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:00,651 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:00,654 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:00,662 [284] [DEBUG] [app] Ending request: urn:request:f441c376-31f0-4fc7-a71d-04ed48aa53b9 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f441c376-31f0-4fc7-a71d-04ed48aa53b9', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:00,663 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:22:00,663 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.032 118 0.032)
gunicorn-web stdout | 2024-04-22 04:22:00,665 [282] [DEBUG] [app] Ending request: urn:request:fc35e082-f483-4ac8-882e-eb71a850e2c4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fc35e082-f483-4ac8-882e-eb71a850e2c4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:00,666 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.035 118 0.035)
gunicorn-web stdout | 2024-04-22 04:22:00,666 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:22:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:22:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:31.842666+00:00 (in 29.999440 seconds)
gcworker stdout | 2024-04-22 04:22:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:31 UTC)" (scheduled at 2024-04-22 04:22:01.842666+00:00)
gcworker stdout | 2024-04-22 04:22:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550121843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:22:01,858 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:22:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:22:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:22:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:22:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:09.033206+00:00 (in 6.997044 seconds)
repositorygcworker stdout | 2024-04-22 04:22:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:02 UTC)" (scheduled at 2024-04-22 04:22:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:22:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:22:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:22:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:12.430973+00:00 (in 9.999500 seconds)
notificationworker stdout | 2024-04-22 04:22:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:12 UTC)" (scheduled at 2024-04-22 04:22:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:22:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:22:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 2, 431741), True, datetime.datetime(2024, 4, 22, 4, 22, 2, 431741), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:22:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:22:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:22:06,145 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:22:07,404 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:22:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:22:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:02.035613+00:00 (in 53.001910 seconds)
repositorygcworker stdout | 2024-04-22 04:22:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:09 UTC)" (scheduled at 2024-04-22 04:22:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:22:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:22:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 9, 34220), True, datetime.datetime(2024, 4, 22, 4, 22, 9, 34220), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:22:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:22:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:22:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:22:09,978 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:22:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:22:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:40.238173+00:00 (in 29.999502 seconds)
autopruneworker stdout | 2024-04-22 04:22:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:40 UTC)" (scheduled at 2024-04-22 04:22:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:22:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759670247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:22:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:22:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:47.245913+00:00 (in 36.996786 seconds)
namespacegcworker stdout | 2024-04-22 04:22:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:10 UTC)" (scheduled at 2024-04-22 04:22:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:22:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:22:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:22:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:22:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:22:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:22:10,998 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:22:11,303 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:22:11,554 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:22:11,929 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:22:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:22.430973+00:00 (in 9.999519 seconds)
notificationworker stdout | 2024-04-22 04:22:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:22 UTC)" (scheduled at 2024-04-22 04:22:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:22:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:22:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 12, 431744), True, datetime.datetime(2024, 4, 22, 4, 22, 12, 431744), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:22:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:22:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:22:12,612 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:22:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:22:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:27.806778+00:00 (in 14.999439 seconds)
quotatotalworker stdout | 2024-04-22 04:22:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:27 UTC)" (scheduled at 2024-04-22 04:22:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:22:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:22:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:22:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:27 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:22:12,965 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:22:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:22:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:17.065745+00:00 (in 2.998196 seconds)
securityworker stdout | 2024-04-22 04:22:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:44 UTC)" (scheduled at 2024-04-22 04:22:14.067050+00:00)
securityworker stdout | 2024-04-22 04:22:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:22:14,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:22:14,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:22:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:22:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:22:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:22:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:22:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:22:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 17, 14, 77465), 1, 2])
securityworker stdout | 2024-04-22 04:22:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 17, 14, 77465), 1, 2])
securityworker stdout | 2024-04-22 04:22:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:22:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:22:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:22:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:22:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:22:14,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:22:14,203 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:22:15,206 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:22:15,632 [282] [DEBUG] [app] Starting request: urn:request:54b3da7c-bb67-4ffe-9dc6-315ec8da9680 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:15,632 [284] [DEBUG] [app] Starting request: urn:request:19a13f3f-d34e-48f4-a636-e8e033617125 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:15,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:15,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:22:15,636 [278] [DEBUG] [app] Starting request: urn:request:ea9518f9-15c8-4e18-981d-24123e2c323f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:15,636 [275] [DEBUG] [app] Starting request: urn:request:9cadcca8-6aa6-4c2a-9b62-90bf6d122fc5 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:15,636 [278] [DEBUG] [app] Ending request: urn:request:ea9518f9-15c8-4e18-981d-24123e2c323f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ea9518f9-15c8-4e18-981d-24123e2c323f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:22:15,636 [275] [DEBUG] [app] Ending request: urn:request:9cadcca8-6aa6-4c2a-9b62-90bf6d122fc5 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:9cadcca8-6aa6-4c2a-9b62-90bf6d122fc5', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:22:15,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:22:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:15,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:15,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:15,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:15,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:15,639 [286] [DEBUG] [app] Starting request: urn:request:9bdc7801-ff7c-425e-9bfd-06208c3a7781 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:15,639 [285] [DEBUG] [app] Starting request: urn:request:9d6db31f-3bf1-42e5-978e-6c5e06127b2f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:15,639 [285] [DEBUG] [app] Ending request: urn:request:9d6db31f-3bf1-42e5-978e-6c5e06127b2f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9d6db31f-3bf1-42e5-978e-6c5e06127b2f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:15,639 [286] [DEBUG] [app] Ending request: urn:request:9bdc7801-ff7c-425e-9bfd-06208c3a7781 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:9bdc7801-ff7c-425e-9bfd-06208c3a7781', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:15,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:15,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:22:15,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:15,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:15,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:15,641 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:15,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:15,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:15,660 [282] [DEBUG] [app] Ending request: urn:request:54b3da7c-bb67-4ffe-9dc6-315ec8da9680 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:54b3da7c-bb67-4ffe-9dc6-315ec8da9680', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:15,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:22:15,660 [284] [DEBUG] [app] Ending request: urn:request:19a13f3f-d34e-48f4-a636-e8e033617125 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:19a13f3f-d34e-48f4-a636-e8e033617125', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:15,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:22:15,660 [284] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:22:15,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:22:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:22:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:44.067050+00:00 (in 27.000743 seconds)
securityworker stdout | 2024-04-22 04:22:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:47 UTC)" (scheduled at 2024-04-22 04:22:17.065745+00:00)
securityworker stdout | 2024-04-22 04:22:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:22:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:22:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:22:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:22:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:22:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:22:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 17, 17, 70587), 1, 2])
securityworker stdout | 2024-04-22 04:22:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 17, 17, 70587), 1, 2])
securityworker stdout | 2024-04-22 04:22:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:22:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:22:17,714 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:22:17,802 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:22:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:22:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:23.046900+00:00 (in 4.999266 seconds)
exportactionlogsworker stdout | 2024-04-22 04:22:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:18 UTC)" (scheduled at 2024-04-22 04:22:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:22:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:22:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:22:18,434 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:22:19,004 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:22:19,895 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:22:20,112 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:22:20,115 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:22:20,118 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:22:21,101 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:22:21,513 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:22:21,684 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:22:21,736 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:22:21,740 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-registry stdout | 2024-04-22 04:22:21,749 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-web stdout | 2024-04-22 04:22:21,753 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:22:21,757 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:22:21,760 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:22:21,762 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:22:21,767 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:22:21,770 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:22:21,774 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:22:21,779 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-registry stdout | 2024-04-22 04:22:21,781 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
gunicorn-web stdout | 2024-04-22 04:22:21,784 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
notificationworker stdout | 2024-04-22 04:22:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:32.430973+00:00 (in 9.999490 seconds)
notificationworker stdout | 2024-04-22 04:22:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:32 UTC)" (scheduled at 2024-04-22 04:22:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:22:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:22:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 22, 431745), True, datetime.datetime(2024, 4, 22, 4, 22, 22, 431745), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:22:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:22:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:22,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:22:22,768 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:22:22,985 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:22:23,023 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:22:23,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:22:23,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:53.045554+00:00 (in 29.998169 seconds)
exportactionlogsworker stdout | 2024-04-22 04:22:23,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:27:23 UTC)" (scheduled at 2024-04-22 04:22:23.046900+00:00)
exportactionlogsworker stdout | 2024-04-22 04:22:23,048 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 22, 23, 47579), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:22:23,061 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 22, 23, 47579), True, datetime.datetime(2024, 4, 22, 4, 22, 23, 47579), 0, 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:22:23,065 [66] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 22, 23, 47579), True, datetime.datetime(2024, 4, 22, 4, 22, 23, 47579), 0, 'exportactionlogs/%', False, datetime.datetime(2024, 4, 22, 4, 22, 23, 47579), 'exportactionlogs/%'])
exportactionlogsworker stdout | 2024-04-22 04:22:23,068 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:22:23,069 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:27:23 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:22:23,937 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:22:23,939 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotaregistrysizeworker stdout | 2024-04-22 04:22:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:22:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:24.738365+00:00 (in 59.999487 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:22:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:24 UTC)" (scheduled at 2024-04-22 04:22:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:22:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:22:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:22:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:22:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:22:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:42.806778+00:00 (in 14.999436 seconds)
quotatotalworker stdout | 2024-04-22 04:22:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:42 UTC)" (scheduled at 2024-04-22 04:22:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:22:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:22:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:22:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:22:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:22:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:58.130973+00:00 (in 29.999554 seconds)
buildlogsarchiver stdout | 2024-04-22 04:22:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:58 UTC)" (scheduled at 2024-04-22 04:22:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:22:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 22, 28, 131794), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:22:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:22:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:22:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:22:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:31.933721+00:00 (in 2.002183 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:29 UTC)" (scheduled at 2024-04-22 04:22:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 29, 931849), True, datetime.datetime(2024, 4, 22, 4, 22, 29, 931849), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:22:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:22:30,632 [284] [DEBUG] [app] Starting request: urn:request:bc2e417e-d5c8-4506-98f3-b95f3db67189 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:30,632 [282] [DEBUG] [app] Starting request: urn:request:63f50223-4cf3-490d-97cf-c88cc381554c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:30,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:22:30,635 [275] [DEBUG] [app] Starting request: urn:request:a4465c4e-ee97-4f05-9a9d-fdf2e83bc791 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:30,635 [283] [DEBUG] [app] Starting request: urn:request:82a7d908-1144-4f47-ae87-f7402c37c3cd (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:30,635 [283] [DEBUG] [app] Ending request: urn:request:82a7d908-1144-4f47-ae87-f7402c37c3cd (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:82a7d908-1144-4f47-ae87-f7402c37c3cd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:22:30,635 [275] [DEBUG] [app] Ending request: urn:request:a4465c4e-ee97-4f05-9a9d-fdf2e83bc791 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:a4465c4e-ee97-4f05-9a9d-fdf2e83bc791', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:22:30,635 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:22:30,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:22:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:30,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:30,638 [284] [DEBUG] [app] Starting request: urn:request:3f0cafed-17d4-4df7-930a-f101fb307fa4 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:30,638 [286] [DEBUG] [app] Starting request: urn:request:656d8955-e81d-49e0-abb8-2ddc381c366e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:30,638 [284] [DEBUG] [app] Ending request: urn:request:3f0cafed-17d4-4df7-930a-f101fb307fa4 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3f0cafed-17d4-4df7-930a-f101fb307fa4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:30,639 [286] [DEBUG] [app] Ending request: urn:request:656d8955-e81d-49e0-abb8-2ddc381c366e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:656d8955-e81d-49e0-abb8-2ddc381c366e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:22:30,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:30,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:22:30,639 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:30,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:30,660 [284] [DEBUG] [app] Ending request: urn:request:bc2e417e-d5c8-4506-98f3-b95f3db67189 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:bc2e417e-d5c8-4506-98f3-b95f3db67189', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:30,660 [282] [DEBUG] [app] Ending request: urn:request:63f50223-4cf3-490d-97cf-c88cc381554c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:63f50223-4cf3-490d-97cf-c88cc381554c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:22:30,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:22:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:22:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:22:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:22:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:01.842666+00:00 (in 29.999440 seconds)
gcworker stdout | 2024-04-22 04:22:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:01 UTC)" (scheduled at 2024-04-22 04:22:31.842666+00:00)
gcworker stdout | 2024-04-22 04:22:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550151843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:22:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:22:31,861 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:22:31,861 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:22:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:22:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:29.931002+00:00 (in 57.996767 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:22:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:31 UTC)" (scheduled at 2024-04-22 04:22:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:22:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:22:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:22:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:38.433394+00:00 (in 6.001898 seconds)
notificationworker stdout | 2024-04-22 04:22:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:42 UTC)" (scheduled at 2024-04-22 04:22:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:22:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:22:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 32, 431836), True, datetime.datetime(2024, 4, 22, 4, 22, 32, 431836), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:22:32,450 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:22:32,450 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:32,450 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:22:36,152 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:22:37,440 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
notificationworker stdout | 2024-04-22 04:22:38,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:38,433 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:42.430973+00:00 (in 3.997108 seconds)
notificationworker stdout | 2024-04-22 04:22:38,433 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:27:38 UTC)" (scheduled at 2024-04-22 04:22:38.433394+00:00)
notificationworker stdout | 2024-04-22 04:22:38,434 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 22, 38, 434094), 'notification/%'])
notificationworker stdout | 2024-04-22 04:22:38,448 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 22, 38, 434094), True, datetime.datetime(2024, 4, 22, 4, 22, 38, 434094), 0, 'notification/%'])
notificationworker stdout | 2024-04-22 04:22:38,452 [77] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 22, 38, 434094), True, datetime.datetime(2024, 4, 22, 4, 22, 38, 434094), 0, 'notification/%', False, datetime.datetime(2024, 4, 22, 4, 22, 38, 434094), 'notification/%'])
notificationworker stdout | 2024-04-22 04:22:38,456 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:38,456 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:27:38 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:22:40,015 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:22:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:22:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:10.238173+00:00 (in 29.999513 seconds)
autopruneworker stdout | 2024-04-22 04:22:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:10 UTC)" (scheduled at 2024-04-22 04:22:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:22:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759700248, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:22:40,254 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:22:40,254 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:22:40,254 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:22:41,035 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:22:41,332 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:22:41,575 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:22:41,966 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:22:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:52.430973+00:00 (in 9.999503 seconds)
notificationworker stdout | 2024-04-22 04:22:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:52 UTC)" (scheduled at 2024-04-22 04:22:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:22:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:22:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 42, 431739), True, datetime.datetime(2024, 4, 22, 4, 22, 42, 431739), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:22:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:22:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:22:52 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:22:42,643 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:22:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:22:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:57.806778+00:00 (in 14.999420 seconds)
quotatotalworker stdout | 2024-04-22 04:22:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:57 UTC)" (scheduled at 2024-04-22 04:22:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:22:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:22:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:22:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:22:57 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:22:42,972 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:22:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:22:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:47.065745+00:00 (in 2.998203 seconds)
securityworker stdout | 2024-04-22 04:22:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:14 UTC)" (scheduled at 2024-04-22 04:22:44.067050+00:00)
securityworker stdout | 2024-04-22 04:22:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:22:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:22:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:22:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:22:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:22:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:22:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:22:44,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:22:44,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:44,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:44,097 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 17, 44, 78039), 1, 2])
securityworker stdout | 2024-04-22 04:22:44,101 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:44,101 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:44,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 17, 44, 78039), 1, 2])
securityworker stdout | 2024-04-22 04:22:44,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:22:44,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:22:44,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:44,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:22:44,106 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:22:44,106 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:22:44,106 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:22:44,107 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:22:44,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:22:44,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:14 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:22:44,226 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:22:45,214 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:22:45,632 [285] [DEBUG] [app] Starting request: urn:request:59d2f807-e5b9-4125-a652-5794a4b87c53 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:45,632 [282] [DEBUG] [app] Starting request: urn:request:52751518-ae28-43d2-a972-771a7cebf34f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:22:45,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:45,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:22:45,635 [275] [DEBUG] [app] Starting request: urn:request:0114c46b-87d6-40b6-b53c-812af32b60e8 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:45,636 [283] [DEBUG] [app] Starting request: urn:request:36244623-d081-466b-b8d9-442826f58bae (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:22:45,636 [275] [DEBUG] [app] Ending request: urn:request:0114c46b-87d6-40b6-b53c-812af32b60e8 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0114c46b-87d6-40b6-b53c-812af32b60e8', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:22:45,636 [283] [DEBUG] [app] Ending request: urn:request:36244623-d081-466b-b8d9-442826f58bae (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:36244623-d081-466b-b8d9-442826f58bae', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:22:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:22:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:22:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:45,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:45,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:22:45,639 [285] [DEBUG] [app] Starting request: urn:request:304431b8-b92e-403f-bfd6-8db516baa509 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:45,639 [284] [DEBUG] [app] Starting request: urn:request:871f57a2-3a3e-40d1-8a69-0bf87f3a1f05 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:22:45,639 [285] [DEBUG] [app] Ending request: urn:request:304431b8-b92e-403f-bfd6-8db516baa509 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:304431b8-b92e-403f-bfd6-8db516baa509', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:45,639 [284] [DEBUG] [app] Ending request: urn:request:871f57a2-3a3e-40d1-8a69-0bf87f3a1f05 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:871f57a2-3a3e-40d1-8a69-0bf87f3a1f05', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:22:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:22:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:22:45,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:22:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:22:45,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:22:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:22:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:45,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:22:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:45,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:22:45,660 [282] [DEBUG] [app] Ending request: urn:request:52751518-ae28-43d2-a972-771a7cebf34f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:52751518-ae28-43d2-a972-771a7cebf34f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:22:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:22:45,661 [285] [DEBUG] [app] Ending request: urn:request:59d2f807-e5b9-4125-a652-5794a4b87c53 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:59d2f807-e5b9-4125-a652-5794a4b87c53', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:22:45,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:22:45,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:22:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:22:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
securityworker stdout | 2024-04-22 04:22:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:22:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:14.067050+00:00 (in 27.000734 seconds)
securityworker stdout | 2024-04-22 04:22:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:17 UTC)" (scheduled at 2024-04-22 04:22:47.065745+00:00)
securityworker stdout | 2024-04-22 04:22:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:22:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:22:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:22:47,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:22:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:22:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:22:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:22:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:10.248668+00:00 (in 23.002252 seconds)
namespacegcworker stdout | 2024-04-22 04:22:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:47 UTC)" (scheduled at 2024-04-22 04:22:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:22:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:22:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 47, 246667), True, datetime.datetime(2024, 4, 22, 4, 22, 47, 246667), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:22:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:22:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:22:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:22:47,736 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:22:47,838 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:22:48,454 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:22:49,020 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:22:49,919 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:22:50,121 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:22:50,123 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:22:50,125 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:22:51,138 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:22:51,525 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:22:51,695 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:22:51,749 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:22:51,752 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:22:51,760 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:22:51,762 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:22:51,769 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:22:51,772 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:22:51,775 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:22:51,779 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:22:51,781 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:22:51,784 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:22:51,787 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:22:51,791 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:22:51,793 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:22:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:22:54.433683+00:00 (in 2.002236 seconds)
notificationworker stdout | 2024-04-22 04:22:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:02 UTC)" (scheduled at 2024-04-22 04:22:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:22:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:22:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 52, 431747), True, datetime.datetime(2024, 4, 22, 4, 22, 52, 431747), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:22:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:22:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:22:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:22:52,789 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:22:53,019 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
exportactionlogsworker stdout | 2024-04-22 04:22:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:22:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:18.047111+00:00 (in 25.001079 seconds)
exportactionlogsworker stdout | 2024-04-22 04:22:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:53 UTC)" (scheduled at 2024-04-22 04:22:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:22:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:22:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 22, 53, 46302), True, datetime.datetime(2024, 4, 22, 4, 22, 53, 46302), 0, 'exportactionlogs/%', 50, 1, 0])
builder stdout | 2024-04-22 04:22:53,059 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
exportactionlogsworker stdout | 2024-04-22 04:22:53,060 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:22:53,060 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:22:53,060 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:53 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:22:53,973 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
repositoryactioncounter stdout | 2024-04-22 04:22:53,975 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
notificationworker stdout | 2024-04-22 04:22:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:22:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:02.430973+00:00 (in 7.996781 seconds)
notificationworker stdout | 2024-04-22 04:22:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:54 UTC)" (scheduled at 2024-04-22 04:22:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:22:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:22:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:23:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:22:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:22:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:12.806778+00:00 (in 14.999448 seconds)
quotatotalworker stdout | 2024-04-22 04:22:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:12 UTC)" (scheduled at 2024-04-22 04:22:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:22:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:22:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:22:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:22:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:22:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:28.130973+00:00 (in 29.999487 seconds)
buildlogsarchiver stdout | 2024-04-22 04:22:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:28 UTC)" (scheduled at 2024-04-22 04:22:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:22:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 22, 58, 131879), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:22:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:22:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:22:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:23:00,633 [282] [DEBUG] [app] Starting request: urn:request:a28bd847-3800-47a9-a52f-0772be2601b4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:00,633 [285] [DEBUG] [app] Starting request: urn:request:79f2f241-4ecc-43ae-8b44-69786a9ad1fa (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:00,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:23:00,636 [275] [DEBUG] [app] Starting request: urn:request:66e88e30-668b-418b-8fdf-0ae0772c762f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:00,636 [283] [DEBUG] [app] Starting request: urn:request:5b6991d5-d29b-43c6-baf2-e5a5178297aa (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:00,636 [275] [DEBUG] [app] Ending request: urn:request:66e88e30-668b-418b-8fdf-0ae0772c762f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:66e88e30-668b-418b-8fdf-0ae0772c762f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:00,636 [283] [DEBUG] [app] Ending request: urn:request:5b6991d5-d29b-43c6-baf2-e5a5178297aa (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:5b6991d5-d29b-43c6-baf2-e5a5178297aa', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:00,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:23:00,637 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:23:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:00,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:00,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:00,640 [284] [DEBUG] [app] Starting request: urn:request:2c3d9d18-92c3-415f-b973-36dd737dbf66 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:00,640 [286] [DEBUG] [app] Starting request: urn:request:a1e6ff1e-0953-47de-8ec4-f38f3cbf5a8e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:00,640 [284] [DEBUG] [app] Ending request: urn:request:2c3d9d18-92c3-415f-b973-36dd737dbf66 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:2c3d9d18-92c3-415f-b973-36dd737dbf66', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:00,640 [286] [DEBUG] [app] Ending request: urn:request:a1e6ff1e-0953-47de-8ec4-f38f3cbf5a8e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:a1e6ff1e-0953-47de-8ec4-f38f3cbf5a8e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:23:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:00,641 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:00,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:00,641 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:00,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:00,641 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:00,650 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:00,650 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:00,662 [285] [DEBUG] [app] Ending request: urn:request:79f2f241-4ecc-43ae-8b44-69786a9ad1fa (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:79f2f241-4ecc-43ae-8b44-69786a9ad1fa', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:00,662 [282] [DEBUG] [app] Ending request: urn:request:a28bd847-3800-47a9-a52f-0772be2601b4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a28bd847-3800-47a9-a52f-0772be2601b4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:00,662 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:00,662 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:00,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:23:00,662 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:23:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:23:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:31.842666+00:00 (in 29.999448 seconds)
gcworker stdout | 2024-04-22 04:23:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:31 UTC)" (scheduled at 2024-04-22 04:23:01.842666+00:00)
gcworker stdout | 2024-04-22 04:23:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550181843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:23:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:23:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:23:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:23:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:23:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:09.033206+00:00 (in 6.997060 seconds)
repositorygcworker stdout | 2024-04-22 04:23:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:02 UTC)" (scheduled at 2024-04-22 04:23:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:23:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:23:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:23:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:12.430973+00:00 (in 9.999521 seconds)
notificationworker stdout | 2024-04-22 04:23:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:12 UTC)" (scheduled at 2024-04-22 04:23:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:23:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:23:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 2, 431740), True, datetime.datetime(2024, 4, 22, 4, 23, 2, 431740), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:23:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:23:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:23:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:23:06,189 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:23:07,461 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:23:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:23:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:26.035364+00:00 (in 17.001666 seconds)
repositorygcworker stdout | 2024-04-22 04:23:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:09 UTC)" (scheduled at 2024-04-22 04:23:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:23:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:23:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 9, 34196), True, datetime.datetime(2024, 4, 22, 4, 23, 9, 34196), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:23:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:23:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:23:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:23:10,045 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:23:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:23:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:40.238173+00:00 (in 29.999519 seconds)
autopruneworker stdout | 2024-04-22 04:23:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:40 UTC)" (scheduled at 2024-04-22 04:23:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:23:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759730247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:23:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:23:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:47.245913+00:00 (in 36.996800 seconds)
namespacegcworker stdout | 2024-04-22 04:23:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:10 UTC)" (scheduled at 2024-04-22 04:23:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:23:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:23:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:23:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:23:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:23:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:23:11,048 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:23:11,363 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:23:11,594 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:23:11,978 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:23:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:22.430973+00:00 (in 9.999443 seconds)
notificationworker stdout | 2024-04-22 04:23:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:22 UTC)" (scheduled at 2024-04-22 04:23:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:23:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:23:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 12, 431824), True, datetime.datetime(2024, 4, 22, 4, 23, 12, 431824), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:23:12,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:23:12,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:23:12,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:23:12,658 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:23:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:23:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:27.806778+00:00 (in 14.999474 seconds)
quotatotalworker stdout | 2024-04-22 04:23:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:27 UTC)" (scheduled at 2024-04-22 04:23:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:23:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:23:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:23:12,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:27 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:23:12,989 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:23:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:23:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:17.065745+00:00 (in 2.998202 seconds)
securityworker stdout | 2024-04-22 04:23:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:44 UTC)" (scheduled at 2024-04-22 04:23:14.067050+00:00)
securityworker stdout | 2024-04-22 04:23:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:23:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:23:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:23:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:23:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:23:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:23:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:23:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:23:14,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:14,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 18, 14, 77864), 1, 2])
securityworker stdout | 2024-04-22 04:23:14,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:14,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:14,101 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 18, 14, 77864), 1, 2])
securityworker stdout | 2024-04-22 04:23:14,105 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:14,105 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:14,105 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:14,105 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:23:14,105 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:23:14,105 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:23:14,105 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:23:14,106 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:14,107 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:23:14,107 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:23:14,262 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:23:15,243 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:23:15,632 [286] [DEBUG] [app] Starting request: urn:request:8208ae0e-1777-4850-b537-f583d1a4ed3c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:15,632 [285] [DEBUG] [app] Starting request: urn:request:a8e362ba-36e0-4109-bc74-0af6d18747d2 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:15,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:23:15,636 [275] [DEBUG] [app] Starting request: urn:request:bbcbb320-f13a-4645-bca8-9ba88f787a3d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:15,636 [281] [DEBUG] [app] Starting request: urn:request:76848ddc-f1a2-4c21-ba83-d528f9e4a870 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:15,636 [275] [DEBUG] [app] Ending request: urn:request:bbcbb320-f13a-4645-bca8-9ba88f787a3d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bbcbb320-f13a-4645-bca8-9ba88f787a3d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:15,636 [281] [DEBUG] [app] Ending request: urn:request:76848ddc-f1a2-4c21-ba83-d528f9e4a870 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:76848ddc-f1a2-4c21-ba83-d528f9e4a870', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:15,637 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:23:15,637 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:23:15,637 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:15,637 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:15,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:15,639 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:15,639 [282] [DEBUG] [app] Starting request: urn:request:963f049f-0239-4c75-b614-807c4f35f4c3 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:15,640 [282] [DEBUG] [app] Ending request: urn:request:963f049f-0239-4c75-b614-807c4f35f4c3 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:963f049f-0239-4c75-b614-807c4f35f4c3', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:23:15,640 [285] [DEBUG] [app] Starting request: urn:request:1b57cf68-0cc7-4184-bb7f-cfa7928ce833 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:15,641 [285] [DEBUG] [app] Ending request: urn:request:1b57cf68-0cc7-4184-bb7f-cfa7928ce833 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:1b57cf68-0cc7-4184-bb7f-cfa7928ce833', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:15,641 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:23:15,641 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:15,641 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:15,642 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:15,642 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:15,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:15,657 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:15,663 [286] [DEBUG] [app] Ending request: urn:request:8208ae0e-1777-4850-b537-f583d1a4ed3c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8208ae0e-1777-4850-b537-f583d1a4ed3c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:15,663 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gunicorn-web stdout | 2024-04-22 04:23:15,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:23:15,668 [285] [DEBUG] [app] Ending request: urn:request:a8e362ba-36e0-4109-bc74-0af6d18747d2 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a8e362ba-36e0-4109-bc74-0af6d18747d2', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:15,668 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:15,668 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.036 118 0.037)
securityworker stdout | 2024-04-22 04:23:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:23:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:44.067050+00:00 (in 27.000806 seconds)
securityworker stdout | 2024-04-22 04:23:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:47 UTC)" (scheduled at 2024-04-22 04:23:17.065745+00:00)
securityworker stdout | 2024-04-22 04:23:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:23:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:23:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:23:17,071 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:23:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:23:17,087 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:23:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:17,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 18, 17, 70780), 1, 2])
securityworker stdout | 2024-04-22 04:23:17,096 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:17,096 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 18, 17, 70780), 1, 2])
securityworker stdout | 2024-04-22 04:23:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:23:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:23:17,772 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:23:17,873 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:23:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:23:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:53.045554+00:00 (in 34.997949 seconds)
exportactionlogsworker stdout | 2024-04-22 04:23:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:18 UTC)" (scheduled at 2024-04-22 04:23:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:23:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:23:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:23:18,490 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:23:19,046 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:23:19,953 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:23:20,129 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:23:20,131 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:23:20,134 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:23:21,152 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:23:21,549 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:23:21,706 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:23:21,760 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:23:21,762 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:23:21,766 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:23:21,771 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:23:21,777 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:23:21,782 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:23:21,785 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:23:21,788 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:23:21,791 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:23:21,794 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:23:21,797 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:23:21,799 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:23:21,803 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:23:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:32.430973+00:00 (in 9.999498 seconds)
notificationworker stdout | 2024-04-22 04:23:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:32 UTC)" (scheduled at 2024-04-22 04:23:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:23:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:23:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 22, 431728), True, datetime.datetime(2024, 4, 22, 4, 23, 22, 431728), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:23:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:23:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:23:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:23:22,816 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:23:23,030 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:23:23,095 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:23:23,983 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:23:24,010 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:23:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:23:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:24.738365+00:00 (in 59.999452 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:23:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:24 UTC)" (scheduled at 2024-04-22 04:23:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:23:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:23:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:23:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:24 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:23:26,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:23:26,035 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:02.035613+00:00 (in 35.999749 seconds)
repositorygcworker stdout | 2024-04-22 04:23:26,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:28:26 UTC)" (scheduled at 2024-04-22 04:23:26.035364+00:00)
repositorygcworker stdout | 2024-04-22 04:23:26,036 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 23, 26, 36267), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:23:26,050 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 23, 26, 36267), True, datetime.datetime(2024, 4, 22, 4, 23, 26, 36267), 0, 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:23:26,054 [91] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 23, 26, 36267), True, datetime.datetime(2024, 4, 22, 4, 23, 26, 36267), 0, 'repositorygc/%', False, datetime.datetime(2024, 4, 22, 4, 23, 26, 36267), 'repositorygc/%'])
repositorygcworker stdout | 2024-04-22 04:23:26,058 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:23:26,058 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:28:26 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:23:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:23:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:42.806778+00:00 (in 14.999443 seconds)
quotatotalworker stdout | 2024-04-22 04:23:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:42 UTC)" (scheduled at 2024-04-22 04:23:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:23:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:23:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:23:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:23:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:23:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:58.130973+00:00 (in 29.999497 seconds)
buildlogsarchiver stdout | 2024-04-22 04:23:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:58 UTC)" (scheduled at 2024-04-22 04:23:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:23:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 23, 28, 131833), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:23:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:23:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:23:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:23:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:30.933445+00:00 (in 1.001962 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:29 UTC)" (scheduled at 2024-04-22 04:23:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 29, 931750), True, datetime.datetime(2024, 4, 22, 4, 23, 29, 931750), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:23:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:23:30,632 [285] [DEBUG] [app] Starting request: urn:request:01e26561-b6ce-41af-9f6a-2c3fd0b879ad (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:30,632 [286] [DEBUG] [app] Starting request: urn:request:36172097-1252-4b18-99ca-c450ccc12ae5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:30,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:23:30,635 [275] [DEBUG] [app] Starting request: urn:request:2bfefdfa-d9ab-440b-9957-e191f55dd32c (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:30,635 [280] [DEBUG] [app] Starting request: urn:request:bd9318fc-2e87-403a-9f5b-5bc2977390cf (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:30,636 [275] [DEBUG] [app] Ending request: urn:request:2bfefdfa-d9ab-440b-9957-e191f55dd32c (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:2bfefdfa-d9ab-440b-9957-e191f55dd32c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:30,636 [280] [DEBUG] [app] Ending request: urn:request:bd9318fc-2e87-403a-9f5b-5bc2977390cf (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:bd9318fc-2e87-403a-9f5b-5bc2977390cf', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:23:30,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:30,638 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:30,638 [286] [DEBUG] [app] Starting request: urn:request:e8058c49-cc6d-4a39-9f78-b64d9b9013d9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:30,639 [286] [DEBUG] [app] Ending request: urn:request:e8058c49-cc6d-4a39-9f78-b64d9b9013d9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:e8058c49-cc6d-4a39-9f78-b64d9b9013d9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:23:30,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:30,640 [284] [DEBUG] [app] Starting request: urn:request:fc456c77-cdbc-4ff8-8bf4-e13841298010 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:30,640 [284] [DEBUG] [app] Ending request: urn:request:fc456c77-cdbc-4ff8-8bf4-e13841298010 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:fc456c77-cdbc-4ff8-8bf4-e13841298010', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:30,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:23:30,641 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:30,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:30,660 [286] [DEBUG] [app] Ending request: urn:request:36172097-1252-4b18-99ca-c450ccc12ae5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:36172097-1252-4b18-99ca-c450ccc12ae5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:30,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:30,660 [285] [DEBUG] [app] Ending request: urn:request:01e26561-b6ce-41af-9f6a-2c3fd0b879ad (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:01e26561-b6ce-41af-9f6a-2c3fd0b879ad', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:30,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:23:30,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,933 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:31.933721+00:00 (in 0.999777 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:28:30 UTC)" (scheduled at 2024-04-22 04:23:30.933445+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,934 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 23, 30, 934168), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,949 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 23, 30, 934168), True, datetime.datetime(2024, 4, 22, 4, 23, 30, 934168), 0, 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,953 [92] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 23, 30, 934168), True, datetime.datetime(2024, 4, 22, 4, 23, 30, 934168), 0, 'secscanv4/%', False, datetime.datetime(2024, 4, 22, 4, 23, 30, 934168), 'secscanv4/%'])
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,957 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:23:30,957 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:28:30 UTC)" executed successfully
gcworker stdout | 2024-04-22 04:23:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:23:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:01.842666+00:00 (in 29.999420 seconds)
gcworker stdout | 2024-04-22 04:23:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:01 UTC)" (scheduled at 2024-04-22 04:23:31.842666+00:00)
gcworker stdout | 2024-04-22 04:23:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550211843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:23:31,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:23:31,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:23:31,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:23:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:23:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:29.931002+00:00 (in 57.996822 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:23:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:31 UTC)" (scheduled at 2024-04-22 04:23:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:23:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:23:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:23:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:42.430973+00:00 (in 9.999495 seconds)
notificationworker stdout | 2024-04-22 04:23:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:42 UTC)" (scheduled at 2024-04-22 04:23:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:23:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:23:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 32, 431746), True, datetime.datetime(2024, 4, 22, 4, 23, 32, 431746), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:23:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:23:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:23:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:23:36,196 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:23:37,469 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:23:40,058 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:23:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:23:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:10.238173+00:00 (in 29.999497 seconds)
autopruneworker stdout | 2024-04-22 04:23:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:10 UTC)" (scheduled at 2024-04-22 04:23:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:23:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759760247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:23:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:23:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:23:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:23:41,074 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:23:41,396 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:23:41,605 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:23:42,015 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:23:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:52.430973+00:00 (in 9.999507 seconds)
notificationworker stdout | 2024-04-22 04:23:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:52 UTC)" (scheduled at 2024-04-22 04:23:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:23:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:23:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 42, 431721), True, datetime.datetime(2024, 4, 22, 4, 23, 42, 431721), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:23:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:23:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:23:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:23:52 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:23:42,669 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:23:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:23:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:57.806778+00:00 (in 14.999452 seconds)
quotatotalworker stdout | 2024-04-22 04:23:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:57 UTC)" (scheduled at 2024-04-22 04:23:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:23:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:23:42,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:23:42,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:23:57 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:23:43,022 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:23:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:23:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:47.065745+00:00 (in 2.998218 seconds)
securityworker stdout | 2024-04-22 04:23:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:14 UTC)" (scheduled at 2024-04-22 04:23:44.067050+00:00)
securityworker stdout | 2024-04-22 04:23:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:23:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:23:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:23:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:23:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:23:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:23:44,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:23:44,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:23:44,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:44,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 18, 44, 77760), 1, 2])
securityworker stdout | 2024-04-22 04:23:44,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:44,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:44,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 18, 44, 77760), 1, 2])
securityworker stdout | 2024-04-22 04:23:44,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:23:44,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:23:44,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:44,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:23:44,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:23:44,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:23:44,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:23:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:23:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:23:44,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:14 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:23:44,293 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:23:45,280 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:23:45,632 [282] [DEBUG] [app] Starting request: urn:request:fd9ad05c-aeb4-4505-8bf6-fdb0e879d269 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:45,632 [286] [DEBUG] [app] Starting request: urn:request:76b94b3a-f0e7-452f-a4ba-edcc38ed6364 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:23:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:23:45,635 [283] [DEBUG] [app] Starting request: urn:request:ac8ef05c-5347-49ba-87af-3eafdef6d3e2 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:45,635 [278] [DEBUG] [app] Starting request: urn:request:ef0fa902-75a6-476b-b44b-c5328b0489a4 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:23:45,636 [283] [DEBUG] [app] Ending request: urn:request:ac8ef05c-5347-49ba-87af-3eafdef6d3e2 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ac8ef05c-5347-49ba-87af-3eafdef6d3e2', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:45,636 [278] [DEBUG] [app] Ending request: urn:request:ef0fa902-75a6-476b-b44b-c5328b0489a4 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:ef0fa902-75a6-476b-b44b-c5328b0489a4', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:23:45,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:23:45,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:23:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:23:45,639 [285] [DEBUG] [app] Starting request: urn:request:674c03a8-157e-4f35-bd9f-a5252b29455e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:45,639 [282] [DEBUG] [app] Starting request: urn:request:71f47f65-d415-439d-98be-a096508006b0 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:23:45,639 [285] [DEBUG] [app] Ending request: urn:request:674c03a8-157e-4f35-bd9f-a5252b29455e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:674c03a8-157e-4f35-bd9f-a5252b29455e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:45,639 [282] [DEBUG] [app] Ending request: urn:request:71f47f65-d415-439d-98be-a096508006b0 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:71f47f65-d415-439d-98be-a096508006b0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:23:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:23:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:23:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:23:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:23:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:45,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:23:45,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:23:45,648 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:45,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:23:45,660 [282] [DEBUG] [app] Ending request: urn:request:fd9ad05c-aeb4-4505-8bf6-fdb0e879d269 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:fd9ad05c-aeb4-4505-8bf6-fdb0e879d269', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:45,660 [286] [DEBUG] [app] Ending request: urn:request:76b94b3a-f0e7-452f-a4ba-edcc38ed6364 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:76b94b3a-f0e7-452f-a4ba-edcc38ed6364', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:23:45,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:23:45,660 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:23:45,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:23:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:23:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:23:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:23:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:14.067050+00:00 (in 27.000768 seconds)
securityworker stdout | 2024-04-22 04:23:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:17 UTC)" (scheduled at 2024-04-22 04:23:47.065745+00:00)
securityworker stdout | 2024-04-22 04:23:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:23:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:23:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:23:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:23:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:23:47,082 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:23:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:23:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:10.248668+00:00 (in 23.002257 seconds)
namespacegcworker stdout | 2024-04-22 04:23:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:47 UTC)" (scheduled at 2024-04-22 04:23:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:23:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:23:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 47, 246785), True, datetime.datetime(2024, 4, 22, 4, 23, 47, 246785), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:23:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:23:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:23:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:23:47,789 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:23:47,909 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:23:48,519 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:23:49,070 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:23:49,989 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:23:50,138 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:23:50,141 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:23:50,143 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:23:51,182 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:23:51,574 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:23:51,716 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:23:51,770 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:23:51,773 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:23:51,776 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:23:51,781 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:23:51,788 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:23:51,792 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:23:51,795 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:23:51,797 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:23:51,800 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:23:51,804 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:23:51,807 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:23:51,809 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:23:51,812 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:23:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:23:54.433683+00:00 (in 2.002190 seconds)
notificationworker stdout | 2024-04-22 04:23:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:02 UTC)" (scheduled at 2024-04-22 04:23:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:23:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:23:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 52, 431775), True, datetime.datetime(2024, 4, 22, 4, 23, 52, 431775), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:23:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:23:52,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:23:52,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:23:52,849 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
exportactionlogsworker stdout | 2024-04-22 04:23:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:23:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:18.047111+00:00 (in 25.001075 seconds)
exportactionlogsworker stdout | 2024-04-22 04:23:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:53 UTC)" (scheduled at 2024-04-22 04:23:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:23:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:23:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 23, 53, 46315), True, datetime.datetime(2024, 4, 22, 4, 23, 53, 46315), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:23:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:23:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:23:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:53 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:23:53,066 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:23:53,108 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:23:54,020 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:23:54,047 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:23:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:23:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:02.430973+00:00 (in 7.996833 seconds)
notificationworker stdout | 2024-04-22 04:23:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:54 UTC)" (scheduled at 2024-04-22 04:23:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:23:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:23:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:24:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:23:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:23:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:12.806778+00:00 (in 14.999438 seconds)
quotatotalworker stdout | 2024-04-22 04:23:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:12 UTC)" (scheduled at 2024-04-22 04:23:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:23:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:23:57,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:23:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:23:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:23:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:28.130973+00:00 (in 29.999488 seconds)
buildlogsarchiver stdout | 2024-04-22 04:23:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:28 UTC)" (scheduled at 2024-04-22 04:23:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:23:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 23, 58, 131885), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:23:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:23:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:23:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:24:00,632 [284] [DEBUG] [app] Starting request: urn:request:196a91d2-5007-4f28-80e7-262b821b194f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:00,632 [282] [DEBUG] [app] Starting request: urn:request:06c688ef-4c0f-4f51-bda7-2c3b3942e9dc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:00,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:24:00,636 [275] [DEBUG] [app] Starting request: urn:request:d1cc97e1-c875-47fe-a17b-28adc657fd6a (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:00,636 [283] [DEBUG] [app] Starting request: urn:request:30a215c4-e332-4372-9ba3-ba008af4d8f9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:00,636 [275] [DEBUG] [app] Ending request: urn:request:d1cc97e1-c875-47fe-a17b-28adc657fd6a (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:d1cc97e1-c875-47fe-a17b-28adc657fd6a', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:24:00,636 [283] [DEBUG] [app] Ending request: urn:request:30a215c4-e332-4372-9ba3-ba008af4d8f9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:30a215c4-e332-4372-9ba3-ba008af4d8f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:24:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:24:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:00,637 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:00,637 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:00,638 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:00,638 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:00,639 [284] [DEBUG] [app] Starting request: urn:request:aaa55d51-3518-4e6f-984f-4b052ea6bd34 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:00,639 [284] [DEBUG] [app] Ending request: urn:request:aaa55d51-3518-4e6f-984f-4b052ea6bd34 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:aaa55d51-3518-4e6f-984f-4b052ea6bd34', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:00,639 [286] [DEBUG] [app] Starting request: urn:request:26434031-0670-40b1-a187-c942f43c6028 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:24:00,640 [286] [DEBUG] [app] Ending request: urn:request:26434031-0670-40b1-a187-c942f43c6028 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:26434031-0670-40b1-a187-c942f43c6028', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:00,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:00,640 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:00,640 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:24:00,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:00,650 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:00,661 [282] [DEBUG] [app] Ending request: urn:request:06c688ef-4c0f-4f51-bda7-2c3b3942e9dc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:06c688ef-4c0f-4f51-bda7-2c3b3942e9dc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:00,661 [284] [DEBUG] [app] Ending request: urn:request:196a91d2-5007-4f28-80e7-262b821b194f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:196a91d2-5007-4f28-80e7-262b821b194f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:00,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:24:00,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:24:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:24:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:31.842666+00:00 (in 29.999390 seconds)
gcworker stdout | 2024-04-22 04:24:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:31 UTC)" (scheduled at 2024-04-22 04:24:01.842666+00:00)
gcworker stdout | 2024-04-22 04:24:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550241843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:24:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:24:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:24:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:24:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:24:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:09.033206+00:00 (in 6.997049 seconds)
repositorygcworker stdout | 2024-04-22 04:24:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:02 UTC)" (scheduled at 2024-04-22 04:24:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:24:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:24:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:24:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:12.430973+00:00 (in 9.999510 seconds)
notificationworker stdout | 2024-04-22 04:24:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:12 UTC)" (scheduled at 2024-04-22 04:24:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:24:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:24:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 2, 431739), True, datetime.datetime(2024, 4, 22, 4, 24, 2, 431739), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:24:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:24:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:24:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:24:06,232 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:24:07,501 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:24:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:24:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:02.035613+00:00 (in 53.001907 seconds)
repositorygcworker stdout | 2024-04-22 04:24:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:09 UTC)" (scheduled at 2024-04-22 04:24:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:24:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:24:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 9, 34163), True, datetime.datetime(2024, 4, 22, 4, 24, 9, 34163), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:24:09,048 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:24:09,048 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:24:09,048 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:24:10,092 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:24:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:24:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:40.238173+00:00 (in 29.999554 seconds)
autopruneworker stdout | 2024-04-22 04:24:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:40 UTC)" (scheduled at 2024-04-22 04:24:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:24:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759790247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:24:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:24:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:47.245913+00:00 (in 36.996811 seconds)
namespacegcworker stdout | 2024-04-22 04:24:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:10 UTC)" (scheduled at 2024-04-22 04:24:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:24:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:24:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:24:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:24:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:24:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:24:11,102 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:24:11,432 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:24:11,641 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:24:12,023 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:24:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:22.430973+00:00 (in 9.999472 seconds)
notificationworker stdout | 2024-04-22 04:24:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:22 UTC)" (scheduled at 2024-04-22 04:24:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:24:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:24:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 12, 431817), True, datetime.datetime(2024, 4, 22, 4, 24, 12, 431817), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:24:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:24:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:24:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:24:12,705 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:24:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:24:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:27.806778+00:00 (in 14.999410 seconds)
quotatotalworker stdout | 2024-04-22 04:24:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:27 UTC)" (scheduled at 2024-04-22 04:24:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:24:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:24:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:24:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:27 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:24:13,032 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:24:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:24:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:17.065745+00:00 (in 2.998151 seconds)
securityworker stdout | 2024-04-22 04:24:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:44 UTC)" (scheduled at 2024-04-22 04:24:14.067050+00:00)
securityworker stdout | 2024-04-22 04:24:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:24:14,070 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:24:14,070 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:24:14,070 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:14,071 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:24:14,071 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:24:14,075 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:24:14,076 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:24:14,089 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:24:14,093 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:14,093 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:14,094 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 19, 14, 76652), 1, 2])
securityworker stdout | 2024-04-22 04:24:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:14,098 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 19, 14, 76652), 1, 2])
securityworker stdout | 2024-04-22 04:24:14,102 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:14,102 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:14,102 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:14,102 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:24:14,102 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:24:14,102 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:24:14,102 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:24:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:24:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:24:14,326 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:24:15,316 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:24:15,631 [285] [DEBUG] [app] Starting request: urn:request:a5bf7a29-ec31-45e0-9fbd-87045553f6bc (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:15,631 [286] [DEBUG] [app] Starting request: urn:request:683b5819-a6ea-4c81-94fc-2520e09e71e5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:24:15,635 [275] [DEBUG] [app] Starting request: urn:request:353cdba1-c8d7-485e-b58e-2e7a565e6e40 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:15,635 [281] [DEBUG] [app] Starting request: urn:request:89d9a2ea-8dcb-4d52-8148-d890cfa1201f (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:15,635 [275] [DEBUG] [app] Ending request: urn:request:353cdba1-c8d7-485e-b58e-2e7a565e6e40 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:353cdba1-c8d7-485e-b58e-2e7a565e6e40', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:24:15,635 [281] [DEBUG] [app] Ending request: urn:request:89d9a2ea-8dcb-4d52-8148-d890cfa1201f (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:89d9a2ea-8dcb-4d52-8148-d890cfa1201f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:24:15,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:24:15,635 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-web stdout | 2024-04-22 04:24:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:15,638 [286] [DEBUG] [app] Starting request: urn:request:d7aeec4e-c22a-4510-b46d-9ed22aa5f6cc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:15,638 [286] [DEBUG] [app] Ending request: urn:request:d7aeec4e-c22a-4510-b46d-9ed22aa5f6cc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d7aeec4e-c22a-4510-b46d-9ed22aa5f6cc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:15,638 [282] [DEBUG] [app] Starting request: urn:request:54c53445-6cad-4ab8-96fe-4aa8ff9898ea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:15,638 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:15,638 [282] [DEBUG] [app] Ending request: urn:request:54c53445-6cad-4ab8-96fe-4aa8ff9898ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:54c53445-6cad-4ab8-96fe-4aa8ff9898ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.000)
gunicorn-web stdout | 2024-04-22 04:24:15,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:24:15,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:15,659 [286] [DEBUG] [app] Ending request: urn:request:683b5819-a6ea-4c81-94fc-2520e09e71e5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:683b5819-a6ea-4c81-94fc-2520e09e71e5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:15,659 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:15,659 [285] [DEBUG] [app] Ending request: urn:request:a5bf7a29-ec31-45e0-9fbd-87045553f6bc (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:a5bf7a29-ec31-45e0-9fbd-87045553f6bc', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:24:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:24:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:24:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:24:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:44.067050+00:00 (in 27.000706 seconds)
securityworker stdout | 2024-04-22 04:24:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:47 UTC)" (scheduled at 2024-04-22 04:24:17.065745+00:00)
securityworker stdout | 2024-04-22 04:24:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:24:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:24:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:24:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:24:17,083 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:24:17,087 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:24:17,091 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:17,091 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:17,092 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 19, 17, 70590), 1, 2])
securityworker stdout | 2024-04-22 04:24:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:17,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 19, 17, 70590), 1, 2])
securityworker stdout | 2024-04-22 04:24:17,100 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:17,100 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:24:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:24:17,805 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:24:17,944 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:24:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:24:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:53.045554+00:00 (in 34.997935 seconds)
exportactionlogsworker stdout | 2024-04-22 04:24:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:18 UTC)" (scheduled at 2024-04-22 04:24:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:24:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:24:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:24:18,541 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:24:19,110 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:24:20,025 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:24:20,146 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:24:20,149 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:24:20,151 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:24:21,208 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:24:21,611 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:24:21,725 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:24:21,780 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:24:21,782 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:24:21,785 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:24:21,791 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:24:21,798 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:24:21,801 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:24:21,803 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:24:21,806 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:24:21,808 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:24:21,811 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:24:21,814 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:24:21,817 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:24:21,820 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:24:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:32.430973+00:00 (in 9.999477 seconds)
notificationworker stdout | 2024-04-22 04:24:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:32 UTC)" (scheduled at 2024-04-22 04:24:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:24:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:24:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 22, 431771), True, datetime.datetime(2024, 4, 22, 4, 24, 22, 431771), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:24:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:24:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:24:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:24:22,867 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:24:23,102 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:24:23,136 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:24:24,057 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:24:24,082 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:24:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:24:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:24.738365+00:00 (in 59.999435 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:24:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:24 UTC)" (scheduled at 2024-04-22 04:24:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:24:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:24:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:24:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:24:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:24:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:42.806778+00:00 (in 14.999456 seconds)
quotatotalworker stdout | 2024-04-22 04:24:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:42 UTC)" (scheduled at 2024-04-22 04:24:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:24:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:24:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:24:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:24:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:24:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:58.130973+00:00 (in 29.999485 seconds)
buildlogsarchiver stdout | 2024-04-22 04:24:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:58 UTC)" (scheduled at 2024-04-22 04:24:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:24:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 24, 28, 131881), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:24:28,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:24:28,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:24:28,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:24:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:31.933721+00:00 (in 2.002267 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:29 UTC)" (scheduled at 2024-04-22 04:24:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 29, 931745), True, datetime.datetime(2024, 4, 22, 4, 24, 29, 931745), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,947 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:24:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:24:30,632 [286] [DEBUG] [app] Starting request: urn:request:7d870b05-fd27-4fe7-94c4-e9167813fa1a (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:30,632 [284] [DEBUG] [app] Starting request: urn:request:4c86c018-dca5-427f-9e58-d4cf7a71549f (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:30,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:30,634 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:24:30,635 [275] [DEBUG] [app] Starting request: urn:request:b1e5df51-60e8-45b2-ba4f-8ba8f19aea98 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:30,635 [277] [DEBUG] [app] Starting request: urn:request:0f10a3e1-03b7-428c-a581-b9551f1505f1 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:30,636 [275] [DEBUG] [app] Ending request: urn:request:b1e5df51-60e8-45b2-ba4f-8ba8f19aea98 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b1e5df51-60e8-45b2-ba4f-8ba8f19aea98', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:24:30,636 [277] [DEBUG] [app] Ending request: urn:request:0f10a3e1-03b7-428c-a581-b9551f1505f1 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0f10a3e1-03b7-428c-a581-b9551f1505f1', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:24:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:24:30,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:30,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:30,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:30,638 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:30,639 [284] [DEBUG] [app] Starting request: urn:request:81680d39-f035-4783-9a6a-5bfa1383ba0c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:30,639 [284] [DEBUG] [app] Ending request: urn:request:81680d39-f035-4783-9a6a-5bfa1383ba0c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:81680d39-f035-4783-9a6a-5bfa1383ba0c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:30,639 [285] [DEBUG] [app] Starting request: urn:request:125a609b-2401-4b48-98e9-c84766be0524 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:24:30,639 [285] [DEBUG] [app] Ending request: urn:request:125a609b-2401-4b48-98e9-c84766be0524 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:125a609b-2401-4b48-98e9-c84766be0524', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:30,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:24:30,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:30,640 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:30,649 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:30,650 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:30,661 [284] [DEBUG] [app] Ending request: urn:request:4c86c018-dca5-427f-9e58-d4cf7a71549f (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4c86c018-dca5-427f-9e58-d4cf7a71549f', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:30,661 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:30,661 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:24:30,662 [286] [DEBUG] [app] Ending request: urn:request:7d870b05-fd27-4fe7-94c4-e9167813fa1a (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:7d870b05-fd27-4fe7-94c4-e9167813fa1a', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:30,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:30,663 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.032)
gcworker stdout | 2024-04-22 04:24:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:24:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:01.842666+00:00 (in 29.999392 seconds)
gcworker stdout | 2024-04-22 04:24:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:01 UTC)" (scheduled at 2024-04-22 04:24:31.842666+00:00)
gcworker stdout | 2024-04-22 04:24:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550271843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:24:31,862 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:24:31,862 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:24:31,863 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:24:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:24:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:29.931002+00:00 (in 57.996828 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:24:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:31 UTC)" (scheduled at 2024-04-22 04:24:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:24:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:24:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:24:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:42.430973+00:00 (in 9.999527 seconds)
notificationworker stdout | 2024-04-22 04:24:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:42 UTC)" (scheduled at 2024-04-22 04:24:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:24:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:24:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 32, 431730), True, datetime.datetime(2024, 4, 22, 4, 24, 32, 431730), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:24:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:24:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:24:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:24:36,245 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:24:37,517 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:24:40,128 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:24:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:24:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:10.238173+00:00 (in 29.999529 seconds)
autopruneworker stdout | 2024-04-22 04:24:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:10 UTC)" (scheduled at 2024-04-22 04:24:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:24:40,251 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759820250, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:24:40,257 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:24:40,257 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:24:40,257 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:24:41,128 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:24:41,442 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:24:41,678 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:24:42,048 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:24:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:52.430973+00:00 (in 9.999498 seconds)
notificationworker stdout | 2024-04-22 04:24:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:52 UTC)" (scheduled at 2024-04-22 04:24:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:24:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:24:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 42, 431741), True, datetime.datetime(2024, 4, 22, 4, 24, 42, 431741), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:24:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:24:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:24:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:24:52 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:24:42,742 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:24:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:24:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:57.806778+00:00 (in 14.999435 seconds)
quotatotalworker stdout | 2024-04-22 04:24:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:57 UTC)" (scheduled at 2024-04-22 04:24:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:24:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:24:42,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:24:42,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:24:57 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:24:43,039 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:24:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:24:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:47.065745+00:00 (in 2.998134 seconds)
securityworker stdout | 2024-04-22 04:24:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:14 UTC)" (scheduled at 2024-04-22 04:24:44.067050+00:00)
securityworker stdout | 2024-04-22 04:24:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:44,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:24:44,071 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:24:44,071 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:24:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:24:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:24:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:24:44,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:24:44,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:24:44,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:44,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:44,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 19, 44, 77324), 1, 2])
securityworker stdout | 2024-04-22 04:24:44,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:44,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:44,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 19, 44, 77324), 1, 2])
securityworker stdout | 2024-04-22 04:24:44,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:24:44,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:24:44,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:44,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:24:44,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:24:44,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:24:44,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:24:44,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:24:44,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:24:44,106 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:14 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:24:44,346 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:24:45,352 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:24:45,632 [282] [DEBUG] [app] Starting request: urn:request:c35a1b60-ecbd-4199-ae4f-b8ba70cedfc4 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:45,632 [285] [DEBUG] [app] Starting request: urn:request:542de0f9-2477-40ff-a9c0-39f0a4fc1e2e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:24:45,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:45,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:24:45,635 [275] [DEBUG] [app] Starting request: urn:request:24dd9c4a-30e5-4a0a-bd8a-a5645600e735 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:45,635 [277] [DEBUG] [app] Starting request: urn:request:6913461a-4bc9-4b1d-99c2-76afab2afa10 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:24:45,636 [275] [DEBUG] [app] Ending request: urn:request:24dd9c4a-30e5-4a0a-bd8a-a5645600e735 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:24dd9c4a-30e5-4a0a-bd8a-a5645600e735', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:24:45,636 [277] [DEBUG] [app] Ending request: urn:request:6913461a-4bc9-4b1d-99c2-76afab2afa10 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:6913461a-4bc9-4b1d-99c2-76afab2afa10', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:24:45,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:24:45,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:45,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:45,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:45,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:45,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:24:45,638 [282] [DEBUG] [app] Starting request: urn:request:b7f3b25a-c104-4cb3-968c-5864f50e807e (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:45,639 [282] [DEBUG] [app] Ending request: urn:request:b7f3b25a-c104-4cb3-968c-5864f50e807e (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:b7f3b25a-c104-4cb3-968c-5864f50e807e', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:45,639 [286] [DEBUG] [app] Starting request: urn:request:cd975351-615b-4758-817f-09a5e24c4c5c (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:24:45,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:45,639 [286] [DEBUG] [app] Ending request: urn:request:cd975351-615b-4758-817f-09a5e24c4c5c (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:cd975351-615b-4758-817f-09a5e24c4c5c', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:24:45,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:24:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:24:45,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:24:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:24:45,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:24:45,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:45,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:24:45,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:45,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:24:45,660 [285] [DEBUG] [app] Ending request: urn:request:542de0f9-2477-40ff-a9c0-39f0a4fc1e2e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:542de0f9-2477-40ff-a9c0-39f0a4fc1e2e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:45,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:45,660 [282] [DEBUG] [app] Ending request: urn:request:c35a1b60-ecbd-4199-ae4f-b8ba70cedfc4 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c35a1b60-ecbd-4199-ae4f-b8ba70cedfc4', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:24:45,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:24:45,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:24:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:24:45,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:24:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
securityworker stdout | 2024-04-22 04:24:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:24:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:14.067050+00:00 (in 27.000760 seconds)
securityworker stdout | 2024-04-22 04:24:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:17 UTC)" (scheduled at 2024-04-22 04:24:47.065745+00:00)
securityworker stdout | 2024-04-22 04:24:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:24:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:24:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:24:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:24:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:24:47,083 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:24:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:24:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:10.248668+00:00 (in 23.002248 seconds)
namespacegcworker stdout | 2024-04-22 04:24:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:47 UTC)" (scheduled at 2024-04-22 04:24:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:24:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:24:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 47, 246650), True, datetime.datetime(2024, 4, 22, 4, 24, 47, 246650), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:24:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:24:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:24:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:24:47,825 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:24:47,975 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:24:48,555 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:24:49,121 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:24:50,061 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:24:50,156 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:24:50,158 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:24:50,161 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:24:51,245 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:24:51,647 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:24:51,734 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:24:51,788 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:24:51,790 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:24:51,793 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:24:51,802 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:24:51,807 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:24:51,810 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:24:51,813 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:24:51,815 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:24:51,819 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:24:51,823 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:24:51,826 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:24:51,828 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:24:51,831 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:24:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:24:54.433683+00:00 (in 2.002245 seconds)
notificationworker stdout | 2024-04-22 04:24:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:02 UTC)" (scheduled at 2024-04-22 04:24:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:24:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:24:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 52, 431697), True, datetime.datetime(2024, 4, 22, 4, 24, 52, 431697), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:24:52,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:24:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:24:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:24:52,877 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
exportactionlogsworker stdout | 2024-04-22 04:24:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:24:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:18.047111+00:00 (in 25.001096 seconds)
exportactionlogsworker stdout | 2024-04-22 04:24:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:53 UTC)" (scheduled at 2024-04-22 04:24:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:24:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:24:53,046 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 24, 53, 46277), True, datetime.datetime(2024, 4, 22, 4, 24, 53, 46277), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:24:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:24:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:24:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:53 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:24:53,124 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:24:53,163 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:24:54,064 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:24:54,117 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:24:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:24:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:02.430973+00:00 (in 7.996831 seconds)
notificationworker stdout | 2024-04-22 04:24:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:54 UTC)" (scheduled at 2024-04-22 04:24:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:24:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:24:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:25:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:24:57,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:24:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:12.806778+00:00 (in 14.999503 seconds)
quotatotalworker stdout | 2024-04-22 04:24:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:12 UTC)" (scheduled at 2024-04-22 04:24:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:24:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:24:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:24:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:24:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:24:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:28.130973+00:00 (in 29.999518 seconds)
buildlogsarchiver stdout | 2024-04-22 04:24:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:28 UTC)" (scheduled at 2024-04-22 04:24:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:24:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 24, 58, 131826), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:24:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:24:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:24:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:25:00,632 [285] [DEBUG] [app] Starting request: urn:request:81ed99ae-4701-4720-ac96-e9c6e4e0d61e (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:00,632 [282] [DEBUG] [app] Starting request: urn:request:57376984-15f2-4395-a9ab-febe4f519064 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:00,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:00,634 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:25:00,635 [275] [DEBUG] [app] Starting request: urn:request:edf347e2-62be-4a73-9e8c-b4e9e2a2baee (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:00,635 [281] [DEBUG] [app] Starting request: urn:request:c5c33794-2b3c-4955-b255-a1cdf394cb53 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:00,635 [275] [DEBUG] [app] Ending request: urn:request:edf347e2-62be-4a73-9e8c-b4e9e2a2baee (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:edf347e2-62be-4a73-9e8c-b4e9e2a2baee', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:00,635 [281] [DEBUG] [app] Ending request: urn:request:c5c33794-2b3c-4955-b255-a1cdf394cb53 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c5c33794-2b3c-4955-b255-a1cdf394cb53', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-registry stdout | 2024-04-22 04:25:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:25:00,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:25:00,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.001)
gunicorn-web stdout | 2024-04-22 04:25:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:00,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:00,638 [282] [DEBUG] [app] Starting request: urn:request:5980cb8e-b1a6-4299-b1a1-104f94d44647 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:00,639 [286] [DEBUG] [app] Starting request: urn:request:43d87459-f869-4995-b3a4-65c4ada0d156 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:00,639 [282] [DEBUG] [app] Ending request: urn:request:5980cb8e-b1a6-4299-b1a1-104f94d44647 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:5980cb8e-b1a6-4299-b1a1-104f94d44647', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:00,639 [286] [DEBUG] [app] Ending request: urn:request:43d87459-f869-4995-b3a4-65c4ada0d156 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:43d87459-f869-4995-b3a4-65c4ada0d156', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:00,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:25:00,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:25:00,639 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:00,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:00,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:00,660 [282] [DEBUG] [app] Ending request: urn:request:57376984-15f2-4395-a9ab-febe4f519064 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:57376984-15f2-4395-a9ab-febe4f519064', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:25:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:25:00,661 [285] [DEBUG] [app] Ending request: urn:request:81ed99ae-4701-4720-ac96-e9c6e4e0d61e (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:81ed99ae-4701-4720-ac96-e9c6e4e0d61e', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:00,661 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:00,662 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.032)
gcworker stdout | 2024-04-22 04:25:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:25:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:31.842666+00:00 (in 29.999499 seconds)
gcworker stdout | 2024-04-22 04:25:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:31 UTC)" (scheduled at 2024-04-22 04:25:01.842666+00:00)
gcworker stdout | 2024-04-22 04:25:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550301843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:25:01,860 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:25:01,860 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:25:01,860 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:25:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:25:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:09.033206+00:00 (in 6.997049 seconds)
repositorygcworker stdout | 2024-04-22 04:25:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:02 UTC)" (scheduled at 2024-04-22 04:25:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:25:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:25:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:25:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:12.430973+00:00 (in 9.999490 seconds)
notificationworker stdout | 2024-04-22 04:25:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:12 UTC)" (scheduled at 2024-04-22 04:25:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:25:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:25:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 2, 431775), True, datetime.datetime(2024, 4, 22, 4, 25, 2, 431775), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:25:02,445 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:25:02,445 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:25:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:25:06,281 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:25:07,533 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:25:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:25:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:02.035613+00:00 (in 53.001919 seconds)
repositorygcworker stdout | 2024-04-22 04:25:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:09 UTC)" (scheduled at 2024-04-22 04:25:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:25:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:25:09,034 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 9, 34173), True, datetime.datetime(2024, 4, 22, 4, 25, 9, 34173), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:25:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:25:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:25:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:25:10,151 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:25:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:25:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:40.238173+00:00 (in 29.999479 seconds)
autopruneworker stdout | 2024-04-22 04:25:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:40 UTC)" (scheduled at 2024-04-22 04:25:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:25:10,247 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759850246, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:25:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:25:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:47.245913+00:00 (in 36.996780 seconds)
namespacegcworker stdout | 2024-04-22 04:25:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:10 UTC)" (scheduled at 2024-04-22 04:25:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:25:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:25:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:25:10,252 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:25:10,252 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:25:10,252 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:25:11,165 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:25:11,479 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:25:11,693 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:25:12,077 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:25:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:22.430973+00:00 (in 9.999529 seconds)
notificationworker stdout | 2024-04-22 04:25:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:22 UTC)" (scheduled at 2024-04-22 04:25:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:25:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:25:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 12, 431674), True, datetime.datetime(2024, 4, 22, 4, 25, 12, 431674), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:25:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:25:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:25:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:22 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:25:12,778 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:25:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:25:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:27.806778+00:00 (in 14.999464 seconds)
quotatotalworker stdout | 2024-04-22 04:25:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:27 UTC)" (scheduled at 2024-04-22 04:25:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:25:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:25:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:25:12,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:27 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:25:13,074 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:25:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:25:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:17.065745+00:00 (in 2.998167 seconds)
securityworker stdout | 2024-04-22 04:25:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:44 UTC)" (scheduled at 2024-04-22 04:25:14.067050+00:00)
securityworker stdout | 2024-04-22 04:25:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:14,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:25:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:25:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:25:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:25:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:25:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:25:14,077 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:25:14,090 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:25:14,094 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:14,094 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:14,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 20, 14, 77606), 1, 2])
securityworker stdout | 2024-04-22 04:25:14,098 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:14,098 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:14,099 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 20, 14, 77606), 1, 2])
securityworker stdout | 2024-04-22 04:25:14,103 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:14,103 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:14,103 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:14,103 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:25:14,103 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:25:14,103 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:25:14,103 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:25:14,104 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:14,104 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:25:14,104 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:25:14,367 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:25:15,388 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:25:15,632 [285] [DEBUG] [app] Starting request: urn:request:4ecc38ba-9a5e-4bdb-9ade-e43b4f45f2cd (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:15,632 [286] [DEBUG] [app] Starting request: urn:request:b7f24c1f-a031-48e1-825d-e8af1cd297e6 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:15,634 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:25:15,635 [281] [DEBUG] [app] Starting request: urn:request:11b816a1-7688-4cfc-8f72-9e0949a97ed0 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:15,635 [280] [DEBUG] [app] Starting request: urn:request:975a20be-9aef-4d94-bc3b-dd8125ab8b07 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:15,635 [281] [DEBUG] [app] Ending request: urn:request:11b816a1-7688-4cfc-8f72-9e0949a97ed0 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:11b816a1-7688-4cfc-8f72-9e0949a97ed0', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:15,636 [280] [DEBUG] [app] Ending request: urn:request:975a20be-9aef-4d94-bc3b-dd8125ab8b07 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:975a20be-9aef-4d94-bc3b-dd8125ab8b07', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:15,636 [281] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:25:15,636 [280] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:25:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:15,639 [284] [DEBUG] [app] Starting request: urn:request:8b299356-8a84-4d41-b173-9b6ae0f757ea (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:15,639 [282] [DEBUG] [app] Starting request: urn:request:3afe800c-90f8-4a9f-80ef-b3812f650a90 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:15,639 [284] [DEBUG] [app] Ending request: urn:request:8b299356-8a84-4d41-b173-9b6ae0f757ea (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:8b299356-8a84-4d41-b173-9b6ae0f757ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:15,639 [282] [DEBUG] [app] Ending request: urn:request:3afe800c-90f8-4a9f-80ef-b3812f650a90 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3afe800c-90f8-4a9f-80ef-b3812f650a90', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:15,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:25:15,640 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:25:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:15,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:15,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:15,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:15,659 [285] [DEBUG] [app] Ending request: urn:request:4ecc38ba-9a5e-4bdb-9ade-e43b4f45f2cd (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:4ecc38ba-9a5e-4bdb-9ade-e43b4f45f2cd', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:15,660 [286] [DEBUG] [app] Ending request: urn:request:b7f24c1f-a031-48e1-825d-e8af1cd297e6 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:b7f24c1f-a031-48e1-825d-e8af1cd297e6', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:15,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:25:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:25:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:25:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:44.067050+00:00 (in 27.000738 seconds)
securityworker stdout | 2024-04-22 04:25:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:47 UTC)" (scheduled at 2024-04-22 04:25:17.065745+00:00)
securityworker stdout | 2024-04-22 04:25:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:25:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:25:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:25:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:25:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:25:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:25:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 20, 17, 70595), 1, 2])
securityworker stdout | 2024-04-22 04:25:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 20, 17, 70595), 1, 2])
securityworker stdout | 2024-04-22 04:25:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:17,100 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:25:17,100 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:25:17,862 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:25:18,011 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
exportactionlogsworker stdout | 2024-04-22 04:25:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:25:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:53.045554+00:00 (in 34.997956 seconds)
exportactionlogsworker stdout | 2024-04-22 04:25:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:18 UTC)" (scheduled at 2024-04-22 04:25:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:25:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:25:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:18 UTC)" executed successfully
globalpromstats stdout | 2024-04-22 04:25:18,591 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:25:19,150 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:25:20,095 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:25:20,163 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:25:20,167 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:25:20,169 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:25:21,276 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:25:21,658 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:25:21,740 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:25:21,795 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:25:21,800 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:25:21,803 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:25:21,811 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:25:21,815 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:25:21,818 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-registry stdout | 2024-04-22 04:25:21,820 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:25:21,824 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:25:21,826 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:25:21,830 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:25:21,833 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:25:21,836 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:25:21,838 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:25:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:32.430973+00:00 (in 9.999547 seconds)
notificationworker stdout | 2024-04-22 04:25:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:32 UTC)" (scheduled at 2024-04-22 04:25:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:25:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:25:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 22, 431695), True, datetime.datetime(2024, 4, 22, 4, 25, 22, 431695), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:25:22,447 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:25:22,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:25:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:25:22,914 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:25:23,136 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:25:23,199 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:25:24,100 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:25:24,138 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:25:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:25:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:24.738365+00:00 (in 59.999490 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:25:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:24 UTC)" (scheduled at 2024-04-22 04:25:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:25:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:25:24,751 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:25:24,751 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:25:27,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:25:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:42.806778+00:00 (in 14.999436 seconds)
quotatotalworker stdout | 2024-04-22 04:25:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:42 UTC)" (scheduled at 2024-04-22 04:25:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:25:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:25:27,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:25:27,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:25:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:25:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:58.130973+00:00 (in 29.999529 seconds)
buildlogsarchiver stdout | 2024-04-22 04:25:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:58 UTC)" (scheduled at 2024-04-22 04:25:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:25:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 25, 28, 131820), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:25:28,145 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:25:28,145 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:25:28,145 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:25:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:31.933721+00:00 (in 2.002265 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:29 UTC)" (scheduled at 2024-04-22 04:25:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 29, 931711), True, datetime.datetime(2024, 4, 22, 4, 25, 29, 931711), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:25:29,946 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:29 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:25:30,632 [282] [DEBUG] [app] Starting request: urn:request:f701ee7b-3413-4970-a0e4-370a980aa69b (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:30,632 [286] [DEBUG] [app] Starting request: urn:request:34b50064-2113-4609-8deb-69e881e8ac29 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:30,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:25:30,635 [277] [DEBUG] [app] Starting request: urn:request:abc06eee-46aa-4cd0-aab2-ad34dafd9360 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:30,635 [278] [DEBUG] [app] Starting request: urn:request:0513c935-ca1c-4757-920c-537e4063b6ea (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:30,635 [277] [DEBUG] [app] Ending request: urn:request:abc06eee-46aa-4cd0-aab2-ad34dafd9360 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:abc06eee-46aa-4cd0-aab2-ad34dafd9360', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:30,635 [278] [DEBUG] [app] Ending request: urn:request:0513c935-ca1c-4757-920c-537e4063b6ea (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0513c935-ca1c-4757-920c-537e4063b6ea', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:30,636 [278] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:25:30,636 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:25:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:30,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:30,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:30,639 [285] [DEBUG] [app] Starting request: urn:request:d9545154-6a97-4097-9941-406c3791353f (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:30,639 [285] [DEBUG] [app] Ending request: urn:request:d9545154-6a97-4097-9941-406c3791353f (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:d9545154-6a97-4097-9941-406c3791353f', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:30,639 [284] [DEBUG] [app] Starting request: urn:request:af9c29b3-14bb-4cd5-84c0-0149f367b558 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:30,639 [284] [DEBUG] [app] Ending request: urn:request:af9c29b3-14bb-4cd5-84c0-0149f367b558 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:af9c29b3-14bb-4cd5-84c0-0149f367b558', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:30,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:25:30,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:30,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.002)
gunicorn-web stdout | 2024-04-22 04:25:30,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:30,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:30,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:30,660 [282] [DEBUG] [app] Ending request: urn:request:f701ee7b-3413-4970-a0e4-370a980aa69b (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:f701ee7b-3413-4970-a0e4-370a980aa69b', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:30,660 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:30,661 [286] [DEBUG] [app] Ending request: urn:request:34b50064-2113-4609-8deb-69e881e8ac29 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:34b50064-2113-4609-8deb-69e881e8ac29', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:30,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:25:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gcworker stdout | 2024-04-22 04:25:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:25:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:01.842666+00:00 (in 29.999484 seconds)
gcworker stdout | 2024-04-22 04:25:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:01 UTC)" (scheduled at 2024-04-22 04:25:31.842666+00:00)
gcworker stdout | 2024-04-22 04:25:31,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550331843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:25:31,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:25:31,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:25:31,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:25:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:25:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:29.931002+00:00 (in 57.996774 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:25:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:31 UTC)" (scheduled at 2024-04-22 04:25:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:25:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:25:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:25:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:42.430973+00:00 (in 9.999502 seconds)
notificationworker stdout | 2024-04-22 04:25:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:42 UTC)" (scheduled at 2024-04-22 04:25:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:25:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:25:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 32, 431739), True, datetime.datetime(2024, 4, 22, 4, 25, 32, 431739), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:25:32,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:25:32,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:25:32,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:25:36,318 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:25:37,545 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
logrotateworker stdout | 2024-04-22 04:25:40,173 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:25:40,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:25:40,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:10.238173+00:00 (in 29.999509 seconds)
autopruneworker stdout | 2024-04-22 04:25:40,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:10 UTC)" (scheduled at 2024-04-22 04:25:40.238173+00:00)
autopruneworker stdout | 2024-04-22 04:25:40,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759880247, None, 1, 0])
autopruneworker stdout | 2024-04-22 04:25:40,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:25:40,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:25:40,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:25:41,201 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:25:41,514 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:25:41,728 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:25:42,093 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:25:42,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:42,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:52.430973+00:00 (in 9.999512 seconds)
notificationworker stdout | 2024-04-22 04:25:42,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:52 UTC)" (scheduled at 2024-04-22 04:25:42.430973+00:00)
notificationworker stdout | 2024-04-22 04:25:42,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:25:42,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 42, 431904), True, datetime.datetime(2024, 4, 22, 4, 25, 42, 431904), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:25:42,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:25:42,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:25:42,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:25:52 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:25:42,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:25:42,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:57.806778+00:00 (in 14.999473 seconds)
quotatotalworker stdout | 2024-04-22 04:25:42,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:57 UTC)" (scheduled at 2024-04-22 04:25:42.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:25:42,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
teamsyncworker stdout | 2024-04-22 04:25:42,815 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
quotatotalworker stdout | 2024-04-22 04:25:42,832 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:25:42,832 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:25:57 UTC)" executed successfully
storagereplication stdout | 2024-04-22 04:25:43,111 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:25:44,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:25:44,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:47.065745+00:00 (in 2.998183 seconds)
securityworker stdout | 2024-04-22 04:25:44,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:14 UTC)" (scheduled at 2024-04-22 04:25:44.067050+00:00)
securityworker stdout | 2024-04-22 04:25:44,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:44,068 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:25:44,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:25:44,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:25:44,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:44,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:25:44,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:25:44,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:25:44,079 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:25:44,102 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:25:44,106 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:44,106 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:44,107 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 20, 44, 78827), 1, 2])
securityworker stdout | 2024-04-22 04:25:44,110 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:44,110 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:44,111 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 20, 44, 78827), 1, 2])
securityworker stdout | 2024-04-22 04:25:44,115 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:25:44,115 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:25:44,115 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:44,115 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:25:44,115 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:25:44,115 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:25:44,115 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:25:44,116 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:25:44,117 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:25:44,117 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:14 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:25:44,404 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:25:45,425 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:25:45,632 [284] [DEBUG] [app] Starting request: urn:request:8eff0f5f-a53f-47f0-9b26-e7746e3e2c52 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:45,632 [286] [DEBUG] [app] Starting request: urn:request:e6d59a44-6195-450d-b32c-9b801ee6efe5 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:25:45,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:45,633 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:25:45,635 [277] [DEBUG] [app] Starting request: urn:request:b867dad0-2752-4912-96d7-fe54f6fcfdad (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:45,635 [275] [DEBUG] [app] Starting request: urn:request:0c8964dd-fb58-48d2-b221-6604728d8d09 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:25:45,635 [277] [DEBUG] [app] Ending request: urn:request:b867dad0-2752-4912-96d7-fe54f6fcfdad (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:b867dad0-2752-4912-96d7-fe54f6fcfdad', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:45,635 [275] [DEBUG] [app] Ending request: urn:request:0c8964dd-fb58-48d2-b221-6604728d8d09 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:0c8964dd-fb58-48d2-b221-6604728d8d09', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:25:45,635 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:25:45,635 [277] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:25:45,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:45,636 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:45 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.002)
gunicorn-web stdout | 2024-04-22 04:25:45,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:45,637 [284] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:25:45,638 [286] [DEBUG] [app] Starting request: urn:request:eb48aa8b-1247-4996-9e34-b2f45d4103f9 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:45,638 [286] [DEBUG] [app] Ending request: urn:request:eb48aa8b-1247-4996-9e34-b2f45d4103f9 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:eb48aa8b-1247-4996-9e34-b2f45d4103f9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:25:45,638 [285] [DEBUG] [app] Starting request: urn:request:db06d13f-9527-43cc-a2aa-393a104d903b (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:25:45,638 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:25:45,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:45,639 [285] [DEBUG] [app] Ending request: urn:request:db06d13f-9527-43cc-a2aa-393a104d903b (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:db06d13f-9527-43cc-a2aa-393a104d903b', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:25:45,639 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:25:45 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:25:45,639 [286] [INFO] [data.database] Connection pooling disabled for postgresql
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:25:45 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:25:45,639 [284] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:25:45,639 [284] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:25:45,647 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:45,648 [284] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:25:45,659 [286] [DEBUG] [app] Ending request: urn:request:e6d59a44-6195-450d-b32c-9b801ee6efe5 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:e6d59a44-6195-450d-b32c-9b801ee6efe5', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:45,659 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:45,659 [284] [DEBUG] [app] Ending request: urn:request:8eff0f5f-a53f-47f0-9b26-e7746e3e2c52 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:8eff0f5f-a53f-47f0-9b26-e7746e3e2c52', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:25:45,659 [284] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:25:45,659 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
gunicorn-web stdout | 2024-04-22 04:25:45,659 [284] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:25:45 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:25:45 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.028 118 0.028)
securityworker stdout | 2024-04-22 04:25:47,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:25:47,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:14.067050+00:00 (in 27.000745 seconds)
securityworker stdout | 2024-04-22 04:25:47,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:17 UTC)" (scheduled at 2024-04-22 04:25:47.065745+00:00)
securityworker stdout | 2024-04-22 04:25:47,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:25:47,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:25:47,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:25:47,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:25:47,082 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:25:47,083 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:17 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:25:47,246 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:25:47,246 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:10.248668+00:00 (in 23.002248 seconds)
namespacegcworker stdout | 2024-04-22 04:25:47,246 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:47 UTC)" (scheduled at 2024-04-22 04:25:47.245913+00:00)
namespacegcworker stdout | 2024-04-22 04:25:47,246 [75] [DEBUG] [workers.queueworker] Getting work item from queue.
namespacegcworker stdout | 2024-04-22 04:25:47,247 [75] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 47, 246647), True, datetime.datetime(2024, 4, 22, 4, 25, 47, 246647), 0, 'namespacegc/%', 50, 1, 0])
namespacegcworker stdout | 2024-04-22 04:25:47,261 [75] [DEBUG] [workers.queueworker] No more work.
namespacegcworker stdout | 2024-04-22 04:25:47,261 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:25:47,261 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:25:47,875 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
queuecleanupworker stdout | 2024-04-22 04:25:48,037 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:25:48,621 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:25:49,186 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:25:50,110 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:25:50,172 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:25:50,175 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:25:50,178 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:25:51,312 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:25:51,695 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:25:51,752 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:25:51,802 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:25:51,809 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:25:51,811 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:25:51,820 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:25:51,822 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:25:51,827 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:25:51,830 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-web stdout | 2024-04-22 04:25:51,834 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-registry stdout | 2024-04-22 04:25:51,837 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:25:51,839 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-web stdout | 2024-04-22 04:25:51,842 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:25:51,844 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:25:51,849 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:25:52,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:52,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:25:54.433683+00:00 (in 2.002206 seconds)
notificationworker stdout | 2024-04-22 04:25:52,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:02 UTC)" (scheduled at 2024-04-22 04:25:52.430973+00:00)
notificationworker stdout | 2024-04-22 04:25:52,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:25:52,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 52, 431892), True, datetime.datetime(2024, 4, 22, 4, 25, 52, 431892), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:25:52,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:25:52,447 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:25:52,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:02 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:25:52,933 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
exportactionlogsworker stdout | 2024-04-22 04:25:53,045 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:25:53,046 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:18.047111+00:00 (in 25.001101 seconds)
exportactionlogsworker stdout | 2024-04-22 04:25:53,046 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:53 UTC)" (scheduled at 2024-04-22 04:25:53.045554+00:00)
exportactionlogsworker stdout | 2024-04-22 04:25:53,046 [66] [DEBUG] [workers.queueworker] Getting work item from queue.
exportactionlogsworker stdout | 2024-04-22 04:25:53,047 [66] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 25, 53, 46272), True, datetime.datetime(2024, 4, 22, 4, 25, 53, 46272), 0, 'exportactionlogs/%', 50, 1, 0])
exportactionlogsworker stdout | 2024-04-22 04:25:53,061 [66] [DEBUG] [workers.queueworker] No more work.
exportactionlogsworker stdout | 2024-04-22 04:25:53,061 [66] [DEBUG] [data.database] Disconnecting from database.
exportactionlogsworker stdout | 2024-04-22 04:25:53,061 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:53 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:25:53,168 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:25:53,235 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:25:54,136 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:25:54,174 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
notificationworker stdout | 2024-04-22 04:25:54,433 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:25:54,434 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:02.430973+00:00 (in 7.996796 seconds)
notificationworker stdout | 2024-04-22 04:25:54,434 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:54 UTC)" (scheduled at 2024-04-22 04:25:54.433683+00:00)
notificationworker stdout | 2024-04-22 04:25:54,434 [77] [DEBUG] [workers.queueworker] Running watchdog.
notificationworker stdout | 2024-04-22 04:25:54,434 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:26:54 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:25:57,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:25:57,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:12.806778+00:00 (in 14.999483 seconds)
quotatotalworker stdout | 2024-04-22 04:25:57,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:26:12 UTC)" (scheduled at 2024-04-22 04:25:57.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:25:57,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:25:57,822 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:25:57,822 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:26:12 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:25:58,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:25:58,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:28.130973+00:00 (in 29.999462 seconds)
buildlogsarchiver stdout | 2024-04-22 04:25:58,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:28 UTC)" (scheduled at 2024-04-22 04:25:58.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:25:58,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 25, 58, 131907), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:25:58,146 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:25:58,146 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:25:58,146 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:28 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:26:00,632 [282] [DEBUG] [app] Starting request: urn:request:c329cd16-1080-4051-8aef-308fe891228d (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:26:00,632 [286] [DEBUG] [app] Starting request: urn:request:98f7edb9-15a8-453d-8c75-2eafdca4dc68 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:26:00,633 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:00,634 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:26:00,635 [275] [DEBUG] [app] Starting request: urn:request:18f1cf77-a99d-4563-af66-7d6f514d39e9 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:26:00,635 [283] [DEBUG] [app] Starting request: urn:request:3bf6df8a-9016-45dc-9111-642fde4872de (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:26:00,636 [275] [DEBUG] [app] Ending request: urn:request:18f1cf77-a99d-4563-af66-7d6f514d39e9 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:18f1cf77-a99d-4563-af66-7d6f514d39e9', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:26:00,636 [283] [DEBUG] [app] Ending request: urn:request:3bf6df8a-9016-45dc-9111-642fde4872de (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:3bf6df8a-9016-45dc-9111-642fde4872de', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:26:00,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:26:00,636 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 162 0.001)
gunicorn-registry stdout | 2024-04-22 04:26:00,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:00 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:26:00,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:00,637 [282] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:00,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:00,639 [286] [DEBUG] [app] Starting request: urn:request:3dd2b94d-3e1a-46ad-afa2-6e393b332cef (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:26:00,639 [286] [DEBUG] [app] Ending request: urn:request:3dd2b94d-3e1a-46ad-afa2-6e393b332cef (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:3dd2b94d-3e1a-46ad-afa2-6e393b332cef', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:26:00,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:26:00,639 [285] [DEBUG] [app] Starting request: urn:request:67fe138a-cbeb-4c40-aa83-5a3a286646db (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:26:00,639 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:00,640 [285] [DEBUG] [app] Ending request: urn:request:67fe138a-cbeb-4c40-aa83-5a3a286646db (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:67fe138a-cbeb-4c40-aa83-5a3a286646db', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:26:00,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:26:00,640 [285] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:00 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:00 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 159 0.001)
gunicorn-web stdout | 2024-04-22 04:26:00,640 [282] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:00,640 [282] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:26:00,649 [282] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:26:00,651 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:26:00,661 [282] [DEBUG] [app] Ending request: urn:request:c329cd16-1080-4051-8aef-308fe891228d (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:c329cd16-1080-4051-8aef-308fe891228d', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:26:00,661 [282] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:26:00,661 [282] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:26:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:26:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:26:00,662 [286] [DEBUG] [app] Ending request: urn:request:98f7edb9-15a8-453d-8c75-2eafdca4dc68 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:98f7edb9-15a8-453d-8c75-2eafdca4dc68', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:26:00,662 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:26:00,662 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:26:00 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:26:00 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.031 118 0.031)
gcworker stdout | 2024-04-22 04:26:01,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:26:01,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:31.842666+00:00 (in 29.999466 seconds)
gcworker stdout | 2024-04-22 04:26:01,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:31 UTC)" (scheduled at 2024-04-22 04:26:01.842666+00:00)
gcworker stdout | 2024-04-22 04:26:01,844 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550361843, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:26:01,859 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:26:01,859 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:26:01,859 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:31 UTC)" executed successfully
repositorygcworker stdout | 2024-04-22 04:26:02,035 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:26:02,036 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:09.033206+00:00 (in 6.997051 seconds)
repositorygcworker stdout | 2024-04-22 04:26:02,036 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:02 UTC)" (scheduled at 2024-04-22 04:26:02.035613+00:00)
repositorygcworker stdout | 2024-04-22 04:26:02,036 [91] [DEBUG] [workers.queueworker] Running watchdog.
repositorygcworker stdout | 2024-04-22 04:26:02,036 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:02 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:26:02,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:26:02,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:12.430973+00:00 (in 9.999520 seconds)
notificationworker stdout | 2024-04-22 04:26:02,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:12 UTC)" (scheduled at 2024-04-22 04:26:02.430973+00:00)
notificationworker stdout | 2024-04-22 04:26:02,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:26:02,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 26, 2, 431838), True, datetime.datetime(2024, 4, 22, 4, 26, 2, 431838), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:26:02,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:26:02,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:26:02,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:12 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:26:06,354 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}
chunkcleanupworker stdout | 2024-04-22 04:26:07,569 [63] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'chunkcleanupworker.py', 'pid': '63'}
repositorygcworker stdout | 2024-04-22 04:26:09,033 [91] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
repositorygcworker stdout | 2024-04-22 04:26:09,033 [91] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:27:02.035613+00:00 (in 53.001899 seconds)
repositorygcworker stdout | 2024-04-22 04:26:09,033 [91] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:09 UTC)" (scheduled at 2024-04-22 04:26:09.033206+00:00)
repositorygcworker stdout | 2024-04-22 04:26:09,034 [91] [DEBUG] [workers.queueworker] Getting work item from queue.
repositorygcworker stdout | 2024-04-22 04:26:09,035 [91] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 26, 9, 34199), True, datetime.datetime(2024, 4, 22, 4, 26, 9, 34199), 0, 'repositorygc/%', 50, 1, 0])
repositorygcworker stdout | 2024-04-22 04:26:09,049 [91] [DEBUG] [workers.queueworker] No more work.
repositorygcworker stdout | 2024-04-22 04:26:09,049 [91] [DEBUG] [data.database] Disconnecting from database.
repositorygcworker stdout | 2024-04-22 04:26:09,049 [91] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:09 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:26:09,244 [96] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
servicekey stdout | 2024-04-22 04:26:09,244 [96] [INFO] [apscheduler.executors.default] Running job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2024-04-22 04:26:09 UTC)" (scheduled at 2024-04-22 04:26:09.243349+00:00)
servicekey stdout | 2024-04-22 04:26:09,245 [96] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 05:21:09.243349+00:00 (in 3299.998068 seconds)
servicekey stdout | 2024-04-22 04:26:09,245 [96] [DEBUG] [__main__] Starting automatic refresh of service key NgveBRHZtfzlcD5o-ezi6BhCP3a8Qf1BtNnKOQyKTwg to new expiration 2024-04-22 06:26:09.245123
servicekey stdout | 2024-04-22 04:26:09,246 [96] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name", "t1"."kid", "t1"."service", "t1"."jwk", "t1"."metadata", "t1"."created_date", "t1"."expiration_date", "t1"."rotation_duration", "t1"."approval_id" FROM "servicekey" AS "t1" LEFT OUTER JOIN "servicekeyapproval" AS "t2" ON ("t1"."approval_id" = "t2"."id") WHERE (("t1"."kid" = %s) AND (NOT ("t1"."expiration_date" <= %s) OR ("t1"."expiration_date" IS %s))) LIMIT %s OFFSET %s', ['NgveBRHZtfzlcD5o-ezi6BhCP3a8Qf1BtNnKOQyKTwg', datetime.datetime(2024, 4, 15, 4, 26, 9, 245860), None, 1, 0])
servicekey stdout | 2024-04-22 04:26:09,260 [96] [DEBUG] [peewee] ('UPDATE "servicekey" SET "name" = %s, "kid" = %s, "service" = %s, "jwk" = %s, "metadata" = %s, "created_date" = %s, "expiration_date" = %s, "rotation_duration" = %s, "approval_id" = %s WHERE ("servicekey"."id" = %s)', ['https://quayregistry-quay-quay-enterprise.apps.whu415az15.qe.azure.devcluster.openshift.com:443', 'NgveBRHZtfzlcD5o-ezi6BhCP3a8Qf1BtNnKOQyKTwg', 'quay', '{"n": "3aqDoIfTXhDmc8ypTfYFc4-0GC5GFxYDSYMF-bC8BAxazPo4shC_EYttAp0rfWPeZCCQLIezwS1AuwGIma2wsAbkoWUykGcd_4yDQYtULVCa0TIWb5MNYJpAj982fPugI8U7DR91P1sG8OFGqlQQcKae8nkMF9MvN45aOxy4tV8xF-h2nrGaVlh3Uy0N7eMb4rgYfCMZSAGF6U98DMH_FD-bly2GyrJc5EzrkqLEc42jmcmH9p_o7OSW06fGCVsHTysWganP703tLHP_mPOHsvCNybJzIgENlIvYCk-vPV6x9E6g8zwZpNUYk9Y_fudnYPUC73_b2wYQPkQtnO9pLw", "e": "AQAB", "kty": "RSA", "kid": "NgveBRHZtfzlcD5o-ezi6BhCP3a8Qf1BtNnKOQyKTwg"}', '{"created_by": "CLI tool"}', datetime.datetime(2024, 4, 22, 3, 33, 0, 481245), datetime.datetime(2024, 4, 22, 6, 26, 9, 245123), None, 1, 1])
servicekey stdout | 2024-04-22 04:26:09,265 [96] [DEBUG] [__main__] Finished automatic refresh of service key NgveBRHZtfzlcD5o-ezi6BhCP3a8Qf1BtNnKOQyKTwg with new expiration 2024-04-22 06:26:09.245123
servicekey stdout | 2024-04-22 04:26:09,266 [96] [DEBUG] [data.database] Disconnecting from database.
servicekey stdout | 2024-04-22 04:26:09,266 [96] [INFO] [apscheduler.executors.default] Job "ServiceKeyWorker._refresh_service_key (trigger: interval[0:55:00], next run at: 2024-04-22 05:21:09 UTC)" executed successfully
logrotateworker stdout | 2024-04-22 04:26:10,189 [72] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'logrotateworker.py', 'pid': '72'}
autopruneworker stdout | 2024-04-22 04:26:10,238 [59] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
autopruneworker stdout | 2024-04-22 04:26:10,238 [59] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:40.238173+00:00 (in 29.999528 seconds)
autopruneworker stdout | 2024-04-22 04:26:10,238 [59] [INFO] [apscheduler.executors.default] Running job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:40 UTC)" (scheduled at 2024-04-22 04:26:10.238173+00:00)
autopruneworker stdout | 2024-04-22 04:26:10,248 [59] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."namespace_id", "t1"."last_ran_ms", "t1"."status" FROM "autoprunetaskstatus" AS "t1" WHERE (("t1"."namespace_id" NOT IN (SELECT "t2"."id" FROM "user" AS "t2" WHERE (("t2"."enabled" = %s) AND ("t2"."id" = "t1"."namespace_id")))) AND (("t1"."last_ran_ms" < %s) OR ("t1"."last_ran_ms" IS %s))) ORDER BY "t1"."last_ran_ms" ASC NULLS first LIMIT %s OFFSET %s FOR UPDATE SKIP LOCKED', [False, 1713759910247, None, 1, 0])
namespacegcworker stdout | 2024-04-22 04:26:10,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:26:10,249 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:30.247852+00:00 (in 19.998733 seconds)
namespacegcworker stdout | 2024-04-22 04:26:10,249 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:10 UTC)" (scheduled at 2024-04-22 04:26:10.248668+00:00)
namespacegcworker stdout | 2024-04-22 04:26:10,249 [75] [DEBUG] [workers.queueworker] Running watchdog.
namespacegcworker stdout | 2024-04-22 04:26:10,249 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:10 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:26:10,253 [59] [INFO] [__main__] no autoprune tasks found, exiting...
autopruneworker stdout | 2024-04-22 04:26:10,253 [59] [DEBUG] [data.database] Disconnecting from database.
autopruneworker stdout | 2024-04-22 04:26:10,253 [59] [INFO] [apscheduler.executors.default] Job "AutoPruneWorker.prune (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:40 UTC)" executed successfully
autopruneworker stdout | 2024-04-22 04:26:11,235 [59] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'autopruneworker.py', 'pid': '59'}
buildlogsarchiver stdout | 2024-04-22 04:26:11,540 [62] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'buildlogsarchiver.py', 'pid': '62'}
quotaregistrysizeworker stdout | 2024-04-22 04:26:11,758 [85] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotaregistrysizeworker.py', 'pid': '85'}
securityscanningnotificationworker stdout | 2024-04-22 04:26:12,129 [92] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityscanningnotificationworker.py', 'pid': '92'}
notificationworker stdout | 2024-04-22 04:26:12,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:26:12,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:22.430973+00:00 (in 9.999459 seconds)
notificationworker stdout | 2024-04-22 04:26:12,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:22 UTC)" (scheduled at 2024-04-22 04:26:12.430973+00:00)
notificationworker stdout | 2024-04-22 04:26:12,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:26:12,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 26, 12, 431809), True, datetime.datetime(2024, 4, 22, 4, 26, 12, 431809), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:26:12,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:26:12,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:26:12,446 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:22 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:26:12,807 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:26:12,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:27.806778+00:00 (in 14.999463 seconds)
quotatotalworker stdout | 2024-04-22 04:26:12,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:26:27 UTC)" (scheduled at 2024-04-22 04:26:12.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:26:12,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:26:12,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:26:12,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:26:27 UTC)" executed successfully
teamsyncworker stdout | 2024-04-22 04:26:12,851 [103] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'teamsyncworker.py', 'pid': '103'}
storagereplication stdout | 2024-04-22 04:26:13,147 [97] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'storagereplication.py', 'pid': '97'}
securityworker stdout | 2024-04-22 04:26:14,067 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:26:14,067 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:17.065745+00:00 (in 2.998213 seconds)
securityworker stdout | 2024-04-22 04:26:14,067 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:44 UTC)" (scheduled at 2024-04-22 04:26:14.067050+00:00)
securityworker stdout | 2024-04-22 04:26:14,067 [95] [DEBUG] [util.locking] Acquiring global lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:26:14,067 [95] [DEBUG] [redis_lock.acquire] Getting 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST' ...
securityworker stdout | 2024-04-22 04:26:14,072 [95] [INFO] [redis_lock.acquire] Got lock for 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:26:14,072 [95] [DEBUG] [redis_lock.refresh.start] Starting thread to refresh lock every 200.0 seconds
securityworker stdout | 2024-04-22 04:26:14,072 [95] [DEBUG] [util.locking] Acquired lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:26:14,072 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:26:14,072 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:26:14,076 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:26:14,078 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:26:14,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:26:14,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:26:14,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:26:14,096 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 21, 14, 77926), 1, 2])
securityworker stdout | 2024-04-22 04:26:14,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:26:14,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:26:14,100 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 21, 14, 77926), 1, 2])
securityworker stdout | 2024-04-22 04:26:14,104 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:26:14,104 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:26:14,104 [95] [DEBUG] [util.locking] Releasing lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:26:14,104 [95] [DEBUG] [redis_lock.refresh.shutdown] Signalling the lock refresher to stop
securityworker stdout | 2024-04-22 04:26:14,104 [95] [DEBUG] [redis_lock.refresh.thread.exit] Exit requested, stopping lock refreshing
securityworker stdout | 2024-04-22 04:26:14,104 [95] [DEBUG] [redis_lock.refresh.exit] Lock refresher has stopped
securityworker stdout | 2024-04-22 04:26:14,104 [95] [DEBUG] [redis_lock.release] Releasing 'lock:SECURITYWORKER_INDEX_RECENT_MANIFEST'.
securityworker stdout | 2024-04-22 04:26:14,105 [95] [DEBUG] [util.locking] Released lock SECURITYWORKER_INDEX_RECENT_MANIFEST
securityworker stdout | 2024-04-22 04:26:14,105 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:26:14,105 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_recent_manifests_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:44 UTC)" executed successfully
servicekey stdout | 2024-04-22 04:26:14,419 [96] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'servicekeyworker.py', 'pid': '96'}
repositorygcworker stdout | 2024-04-22 04:26:15,461 [91] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositorygcworker.py', 'pid': '91'}
gunicorn-web stdout | 2024-04-22 04:26:15,632 [286] [DEBUG] [app] Starting request: urn:request:13109363-b71a-4dd1-9451-f5c335560378 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:26:15,632 [285] [DEBUG] [app] Starting request: urn:request:99b90bb5-26a2-45d9-9f62-a214a2040798 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:26:15,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:15,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:26:15,635 [275] [DEBUG] [app] Starting request: urn:request:56f56968-ff3f-41e2-8523-78e46976ac27 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:26:15,635 [279] [DEBUG] [app] Starting request: urn:request:cb87e37b-778e-40d2-b98f-db3b5dc0ab96 (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:26:15,635 [275] [DEBUG] [app] Ending request: urn:request:56f56968-ff3f-41e2-8523-78e46976ac27 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:56f56968-ff3f-41e2-8523-78e46976ac27', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:26:15,635 [279] [DEBUG] [app] Ending request: urn:request:cb87e37b-778e-40d2-b98f-db3b5dc0ab96 (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:cb87e37b-778e-40d2-b98f-db3b5dc0ab96', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:26:15,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:26:15,636 [279] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:15 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:26:15,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:15,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:15,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:15,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:15,638 [282] [DEBUG] [app] Starting request: urn:request:48da9987-4209-45cb-bdbc-87af64f52bce (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:26:15,639 [282] [DEBUG] [app] Ending request: urn:request:48da9987-4209-45cb-bdbc-87af64f52bce (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:48da9987-4209-45cb-bdbc-87af64f52bce', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:26:15,639 [284] [DEBUG] [app] Starting request: urn:request:351baaaa-5c66-4533-aedc-69ce1eef49bc (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:26:15,639 [282] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:26:15,639 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:15,639 [284] [DEBUG] [app] Ending request: urn:request:351baaaa-5c66-4533-aedc-69ce1eef49bc (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:351baaaa-5c66-4533-aedc-69ce1eef49bc', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:26:15,639 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:26:15,640 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:15 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:26:15,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:15 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.002)
gunicorn-web stdout | 2024-04-22 04:26:15,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:26:15,648 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:26:15,648 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:26:15,659 [285] [DEBUG] [app] Ending request: urn:request:99b90bb5-26a2-45d9-9f62-a214a2040798 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:99b90bb5-26a2-45d9-9f62-a214a2040798', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:26:15,659 [285] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:26:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
gunicorn-web stdout | 2024-04-22 04:26:15,660 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:26:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:26:15,660 [286] [DEBUG] [app] Ending request: urn:request:13109363-b71a-4dd1-9451-f5c335560378 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:13109363-b71a-4dd1-9451-f5c335560378', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:26:15,660 [286] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:26:15,660 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:26:15 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:26:15 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.029 118 0.029)
securityworker stdout | 2024-04-22 04:26:17,065 [95] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityworker stdout | 2024-04-22 04:26:17,066 [95] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:44.067050+00:00 (in 27.000778 seconds)
securityworker stdout | 2024-04-22 04:26:17,066 [95] [INFO] [apscheduler.executors.default] Running job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:47 UTC)" (scheduled at 2024-04-22 04:26:17.065745+00:00)
securityworker stdout | 2024-04-22 04:26:17,066 [95] [DEBUG] [util.secscan.v4.api] generated jwt for security scanner request
securityworker stdout | 2024-04-22 04:26:17,066 [95] [DEBUG] [util.secscan.v4.api] GETing security URL http://quayregistry-clair-app.quay-enterprise.svc.cluster.local/indexer/api/v1/index_state
securityworker stdout | 2024-04-22 04:26:17,069 [95] [DEBUG] [urllib3.connectionpool] http://quayregistry-clair-app.quay-enterprise.svc.cluster.local:80 "GET /indexer/api/v1/index_state HTTP/1.1" 200 None
securityworker stdout | 2024-04-22 04:26:17,070 [95] [DEBUG] [peewee] ('SELECT Max("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:26:17,082 [95] [DEBUG] [peewee] ('SELECT Min("t1"."id") FROM "manifest" AS "t1"', [])
securityworker stdout | 2024-04-22 04:26:17,086 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" LEFT OUTER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE ((("t2"."id" IS %s) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [None, 1, 2])
securityworker stdout | 2024-04-22 04:26:17,090 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:26:17,090 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:26:17,091 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((("t2"."index_status" = %s) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-1, datetime.datetime(2024, 4, 22, 4, 21, 17, 70549), 1, 2])
securityworker stdout | 2024-04-22 04:26:17,095 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:26:17,095 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:26:17,095 [95] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."repository_id", "t1"."digest", "t1"."media_type_id", "t1"."manifest_bytes", "t1"."config_media_type", "t1"."layers_compressed_size", "t2"."id", "t2"."manifest_id", "t2"."repository_id", "t2"."index_status", "t2"."error_json", "t2"."last_indexed", "t2"."indexer_hash", "t2"."indexer_version", "t2"."metadata_json" FROM "manifest" AS "t1" INNER JOIN "manifestsecuritystatus" AS "t2" ON ("t2"."manifest_id" = "t1"."id") WHERE (((((("t2"."index_status" != %s) AND ("t2"."index_status" != %s)) AND ("t2"."indexer_hash" != %s)) AND ("t2"."last_indexed" < %s)) AND ("t1"."id" >= %s)) AND ("t1"."id" < %s)) ORDER BY "t1"."id"', [-2, -3, '746c21e6e713e5208118f7d54c52e391', datetime.datetime(2024, 4, 22, 4, 21, 17, 70549), 1, 2])
securityworker stdout | 2024-04-22 04:26:17,099 [95] [INFO] [util.migrate.allocator] No candidates, marking entire block completed 1-2
securityworker stdout | 2024-04-22 04:26:17,099 [95] [INFO] [util.migrate.allocator] No more work
securityworker stdout | 2024-04-22 04:26:17,099 [95] [DEBUG] [data.database] Disconnecting from database.
securityworker stdout | 2024-04-22 04:26:17,099 [95] [INFO] [apscheduler.executors.default] Job "SecurityWorker._index_in_scanner (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:47 UTC)" executed successfully
expiredappspecifictokenworker stdout | 2024-04-22 04:26:17,897 [65] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'expiredappspecifictokenworker.py', 'pid': '65'}
exportactionlogsworker stdout | 2024-04-22 04:26:18,047 [66] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
exportactionlogsworker stdout | 2024-04-22 04:26:18,047 [66] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:53.045554+00:00 (in 34.997977 seconds)
exportactionlogsworker stdout | 2024-04-22 04:26:18,047 [66] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:18 UTC)" (scheduled at 2024-04-22 04:26:18.047111+00:00)
exportactionlogsworker stdout | 2024-04-22 04:26:18,047 [66] [DEBUG] [workers.queueworker] Running watchdog.
exportactionlogsworker stdout | 2024-04-22 04:26:18,047 [66] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:18 UTC)" executed successfully
queuecleanupworker stdout | 2024-04-22 04:26:18,070 [84] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'queuecleanupworker.py', 'pid': '84'}
globalpromstats stdout | 2024-04-22 04:26:18,637 [68] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'globalpromstats.py', 'pid': '68'}
exportactionlogsworker stdout | 2024-04-22 04:26:19,224 [66] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'exportactionlogsworker.py', 'pid': '66'}
gcworker stdout | 2024-04-22 04:26:20,146 [67] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'gcworker.py', 'pid': '67'}
gunicorn-secscan stdout | 2024-04-22 04:26:20,180 [265] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '265'}
gunicorn-secscan stdout | 2024-04-22 04:26:20,183 [266] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '266'}
gunicorn-secscan stdout | 2024-04-22 04:26:20,185 [70] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'secscan:application', 'pid': '70'}
securityworker stdout | 2024-04-22 04:26:21,350 [95] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'securityworker.py', 'pid': '95'}
manifestbackfillworker stdout | 2024-04-22 04:26:21,731 [73] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'manifestbackfillworker.py', 'pid': '73'}
gunicorn-registry stdout | 2024-04-22 04:26:21,764 [281] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '281'}
gunicorn-registry stdout | 2024-04-22 04:26:21,814 [69] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '69'}
gunicorn-registry stdout | 2024-04-22 04:26:21,817 [278] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '278'}
gunicorn-web stdout | 2024-04-22 04:26:21,821 [286] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '286'}
gunicorn-registry stdout | 2024-04-22 04:26:21,828 [275] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '275'}
gunicorn-registry stdout | 2024-04-22 04:26:21,831 [283] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '283'}
gunicorn-registry stdout | 2024-04-22 04:26:21,834 [279] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '279'}
gunicorn-registry stdout | 2024-04-22 04:26:21,839 [277] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '277'}
gunicorn-web stdout | 2024-04-22 04:26:21,841 [282] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '282'}
gunicorn-web stdout | 2024-04-22 04:26:21,848 [285] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '285'}
gunicorn-registry stdout | 2024-04-22 04:26:21,851 [276] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '276'}
gunicorn-web stdout | 2024-04-22 04:26:21,853 [71] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '71'}
gunicorn-web stdout | 2024-04-22 04:26:21,856 [284] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'web:application', 'pid': '284'}
gunicorn-registry stdout | 2024-04-22 04:26:21,859 [280] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'registry:application', 'pid': '280'}
notificationworker stdout | 2024-04-22 04:26:22,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:26:22,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:32.430973+00:00 (in 9.999532 seconds)
notificationworker stdout | 2024-04-22 04:26:22,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:32 UTC)" (scheduled at 2024-04-22 04:26:22.430973+00:00)
notificationworker stdout | 2024-04-22 04:26:22,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:26:22,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 26, 22, 431708), True, datetime.datetime(2024, 4, 22, 4, 26, 22, 431708), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:26:22,446 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:26:22,446 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:26:22,447 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:32 UTC)" executed successfully
blobuploadcleanupworker stdout | 2024-04-22 04:26:22,948 [60] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'blobuploadcleanupworker.py', 'pid': '60'}
namespacegcworker stdout | 2024-04-22 04:26:23,204 [75] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'namespacegcworker.py', 'pid': '75'}
builder stdout | 2024-04-22 04:26:23,245 [61] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'builder.py', 'pid': '61'}
repositoryactioncounter stdout | 2024-04-22 04:26:24,173 [90] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'repositoryactioncounter.py', 'pid': '90'}
quotatotalworker stdout | 2024-04-22 04:26:24,210 [86] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'quotatotalworker.py', 'pid': '86'}
quotaregistrysizeworker stdout | 2024-04-22 04:26:24,738 [85] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotaregistrysizeworker stdout | 2024-04-22 04:26:24,738 [85] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:27:24.738365+00:00 (in 59.999489 seconds)
quotaregistrysizeworker stdout | 2024-04-22 04:26:24,739 [85] [INFO] [apscheduler.executors.default] Running job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:24 UTC)" (scheduled at 2024-04-22 04:26:24.738365+00:00)
quotaregistrysizeworker stdout | 2024-04-22 04:26:24,739 [85] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."size_bytes", "t1"."running", "t1"."queued", "t1"."completed_ms" FROM "quotaregistrysize" AS "t1" LIMIT %s OFFSET %s', [1, 0])
quotaregistrysizeworker stdout | 2024-04-22 04:26:24,752 [85] [DEBUG] [data.database] Disconnecting from database.
quotaregistrysizeworker stdout | 2024-04-22 04:26:24,752 [85] [INFO] [apscheduler.executors.default] Job "QuotaRegistrySizeWorker._calculate_registry_size (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:24 UTC)" executed successfully
quotatotalworker stdout | 2024-04-22 04:26:27,806 [86] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
quotatotalworker stdout | 2024-04-22 04:26:27,807 [86] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:42.806778+00:00 (in 14.999510 seconds)
quotatotalworker stdout | 2024-04-22 04:26:27,807 [86] [INFO] [apscheduler.executors.default] Running job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:26:42 UTC)" (scheduled at 2024-04-22 04:26:27.806778+00:00)
quotatotalworker stdout | 2024-04-22 04:26:27,808 [86] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."uuid", "t1"."username", "t1"."password_hash", "t1"."email", "t1"."verified", "t1"."stripe_id", "t1"."organization", "t1"."robot", "t1"."invoice_email", "t1"."invalid_login_attempts", "t1"."last_invalid_login", "t1"."removed_tag_expiration_s", "t1"."enabled", "t1"."invoice_email_address", "t1"."given_name", "t1"."family_name", "t1"."company", "t1"."location", "t1"."maximum_queued_builds_count", "t1"."creation_date", "t1"."last_accessed" FROM "user" AS "t1" WHERE ((NOT EXISTS(SELECT "t2"."id" FROM "quotanamespacesize" AS "t2" WHERE (("t2"."namespace_user_id" = "t1"."id") AND ("t2"."backfill_start_ms" IS NOT %s))) AND ("t1"."enabled" = %s)) AND ("t1"."robot" = %s)) LIMIT %s', [None, True, False, 100])
quotatotalworker stdout | 2024-04-22 04:26:27,821 [86] [DEBUG] [data.database] Disconnecting from database.
quotatotalworker stdout | 2024-04-22 04:26:27,821 [86] [INFO] [apscheduler.executors.default] Job "QuotaTotalWorker.backfill (trigger: interval[0:00:15], next run at: 2024-04-22 04:26:42 UTC)" executed successfully
buildlogsarchiver stdout | 2024-04-22 04:26:28,131 [62] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
buildlogsarchiver stdout | 2024-04-22 04:26:28,131 [62] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:58.130973+00:00 (in 29.999525 seconds)
buildlogsarchiver stdout | 2024-04-22 04:26:28,131 [62] [INFO] [apscheduler.executors.default] Running job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:58 UTC)" (scheduled at 2024-04-22 04:26:28.130973+00:00)
buildlogsarchiver stdout | 2024-04-22 04:26:28,132 [62] [DEBUG] [peewee] ('SELECT "candidates"."id" FROM (SELECT "t1"."id" FROM "repositorybuild" AS "t1" WHERE ((("t1"."phase" IN (%s, %s, %s)) OR ("t1"."started" < %s)) AND ("t1"."logs_archived" = %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', ['complete', 'error', 'cancelled', datetime.datetime(2024, 4, 7, 4, 26, 28, 131808), False, 50, 1, 0])
buildlogsarchiver stdout | 2024-04-22 04:26:28,147 [62] [DEBUG] [__main__] No more builds to archive
buildlogsarchiver stdout | 2024-04-22 04:26:28,147 [62] [DEBUG] [data.database] Disconnecting from database.
buildlogsarchiver stdout | 2024-04-22 04:26:28,147 [62] [INFO] [apscheduler.executors.default] Job "ArchiveBuildLogsWorker._archive_redis_buildlogs (trigger: interval[0:00:30], next run at: 2024-04-22 04:26:58 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,931 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,931 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:31.933721+00:00 (in 2.002240 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,931 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:29 UTC)" (scheduled at 2024-04-22 04:26:29.931002+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,931 [92] [DEBUG] [workers.queueworker] Getting work item from queue.
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,932 [92] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 26, 29, 931759), True, datetime.datetime(2024, 4, 22, 4, 26, 29, 931759), 0, 'secscanv4/%', 50, 1, 0])
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,946 [92] [DEBUG] [workers.queueworker] No more work.
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,946 [92] [DEBUG] [data.database] Disconnecting from database.
securityscanningnotificationworker stdout | 2024-04-22 04:26:29,947 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:29 UTC)" executed successfully
namespacegcworker stdout | 2024-04-22 04:26:30,248 [75] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
namespacegcworker stdout | 2024-04-22 04:26:30,248 [75] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:47.245913+00:00 (in 16.997537 seconds)
namespacegcworker stdout | 2024-04-22 04:26:30,248 [75] [INFO] [apscheduler.executors.default] Running job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:31:30 UTC)" (scheduled at 2024-04-22 04:26:30.247852+00:00)
namespacegcworker stdout | 2024-04-22 04:26:30,249 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [False, datetime.datetime(2024, 4, 22, 4, 26, 30, 248751), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:26:30,262 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 26, 30, 248751), True, datetime.datetime(2024, 4, 22, 4, 26, 30, 248751), 0, 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:26:30,266 [75] [DEBUG] [peewee] ('SELECT COUNT(1) FROM (SELECT DISTINCT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) AND NOT ("t1"."queue_name" IN (SELECT "t1"."queue_name" FROM "queueitem" AS "t1" WHERE ((("t1"."available" = %s) AND ("t1"."processing_expires" > %s)) AND ("t1"."queue_name" ILIKE %s)))))) AS "_wrapped"', [datetime.datetime(2024, 4, 22, 4, 26, 30, 248751), True, datetime.datetime(2024, 4, 22, 4, 26, 30, 248751), 0, 'namespacegc/%', False, datetime.datetime(2024, 4, 22, 4, 26, 30, 248751), 'namespacegc/%'])
namespacegcworker stdout | 2024-04-22 04:26:30,270 [75] [DEBUG] [data.database] Disconnecting from database.
namespacegcworker stdout | 2024-04-22 04:26:30,270 [75] [INFO] [apscheduler.executors.default] Job "QueueWorker.update_queue_metrics (trigger: interval[0:05:00], next run at: 2024-04-22 04:31:30 UTC)" executed successfully
gunicorn-web stdout | 2024-04-22 04:26:30,632 [285] [DEBUG] [app] Starting request: urn:request:ed1f5886-2a03-4338-8716-d15e52c78365 (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:26:30,632 [286] [DEBUG] [app] Starting request: urn:request:534c038f-c119-4438-9bcd-c414a05ffc4c (/health/instance) {'X-Forwarded-For': '10.129.2.2'}
gunicorn-web stdout | 2024-04-22 04:26:30,633 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:30,633 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-registry stdout | 2024-04-22 04:26:30,635 [275] [DEBUG] [app] Starting request: urn:request:e946ae40-eb9b-40f1-b105-1acf8dc22e9d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:26:30,635 [283] [DEBUG] [app] Starting request: urn:request:c3a2d546-e1d4-4f2b-bf7d-df8b64774d9d (/v1/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-registry stdout | 2024-04-22 04:26:30,635 [275] [DEBUG] [app] Ending request: urn:request:e946ae40-eb9b-40f1-b105-1acf8dc22e9d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:e946ae40-eb9b-40f1-b105-1acf8dc22e9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:26:30,635 [283] [DEBUG] [app] Ending request: urn:request:c3a2d546-e1d4-4f2b-bf7d-df8b64774d9d (/v1/_internal_ping) {'endpoint': 'v1.internal_ping', 'request_id': 'urn:request:c3a2d546-e1d4-4f2b-bf7d-df8b64774d9d', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/v1/_internal_ping', 'path': '/v1/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'ccc1cb7b', 'user-agent': 'python-requests/2.31.0'}
gunicorn-registry stdout | 2024-04-22 04:26:30,636 [275] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
gunicorn-registry stdout | 2024-04-22 04:26:30,636 [283] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0"
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:30 +0000] "GET /v1/_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.002 162 0.002)
gunicorn-web stdout | 2024-04-22 04:26:30,636 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:30,636 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /v1/_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:30,637 [286] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:30,637 [285] [DEBUG] [urllib3.connectionpool] Resetting dropped connection: localhost
gunicorn-web stdout | 2024-04-22 04:26:30,639 [286] [DEBUG] [app] Starting request: urn:request:abf28224-a5ab-4843-90cc-50a7710c26fd (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:26:30,639 [284] [DEBUG] [app] Starting request: urn:request:06d1c0da-6143-4270-80ca-a14d89e53994 (/_internal_ping) {'X-Forwarded-For': '127.0.0.1'}
gunicorn-web stdout | 2024-04-22 04:26:30,639 [286] [DEBUG] [app] Ending request: urn:request:abf28224-a5ab-4843-90cc-50a7710c26fd (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:abf28224-a5ab-4843-90cc-50a7710c26fd', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
gunicorn-web stdout | 2024-04-22 04:26:30,639 [284] [DEBUG] [app] Ending request: urn:request:06d1c0da-6143-4270-80ca-a14d89e53994 (/_internal_ping) {'endpoint': 'web.internal_ping', 'request_id': 'urn:request:06d1c0da-6143-4270-80ca-a14d89e53994', 'remote_addr': '127.0.0.1', 'http_method': 'GET', 'original_url': 'http://localhost/_internal_ping', 'path': '/_internal_ping', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'python-requests/2.31.0'}
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
nginx stdout | 127.0.0.1 (-) - - [22/Apr/2024:04:26:30 +0000] "GET /_internal_ping HTTP/1.1" 200 4 "-" "python-requests/2.31.0" (0.001 159 0.001)
gunicorn-web stdout | 2024-04-22 04:26:30,639 [286] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:26:30,639 [284] [INFO] [gunicorn.access] 127.0.0.1 - - [22/Apr/2024:04:26:30 +0000] "GET /_internal_ping HTTP/1.0" 200 4 "-" "python-requests/2.31.0"
gunicorn-web stdout | 2024-04-22 04:26:30,640 [285] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:30,640 [286] [DEBUG] [urllib3.connectionpool] http://localhost:8080 "GET /_internal_ping HTTP/1.1" 200 4
gunicorn-web stdout | 2024-04-22 04:26:30,640 [285] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:26:30,640 [286] [INFO] [data.database] Connection pooling disabled for postgresql
gunicorn-web stdout | 2024-04-22 04:26:30,649 [285] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:26:30,649 [286] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."name" FROM "teamrole" AS "t1" LIMIT %s', [1])
gunicorn-web stdout | 2024-04-22 04:26:30,660 [285] [DEBUG] [app] Ending request: urn:request:ed1f5886-2a03-4338-8716-d15e52c78365 (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:ed1f5886-2a03-4338-8716-d15e52c78365', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:26:30,660 [285] [DEBUG] [data.database] Disconnecting from database.
gunicorn-web stdout | 2024-04-22 04:26:30,661 [286] [DEBUG] [app] Ending request: urn:request:534c038f-c119-4438-9bcd-c414a05ffc4c (/health/instance) {'endpoint': 'web.instance_health', 'request_id': 'urn:request:534c038f-c119-4438-9bcd-c414a05ffc4c', 'remote_addr': '10.129.2.2', 'http_method': 'GET', 'original_url': 'http://10.129.2.23/health/instance', 'path': '/health/instance', 'parameters': {}, 'json_body': None, 'confsha': 'c23a8dd5', 'user-agent': 'kube-probe/1.28'}
gunicorn-web stdout | 2024-04-22 04:26:30,661 [285] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:26:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gunicorn-web stdout | 2024-04-22 04:26:30,661 [286] [DEBUG] [data.database] Disconnecting from database.
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:26:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
nginx stdout | 10.129.2.2 (-) - - [22/Apr/2024:04:26:30 +0000] "GET /health/instance HTTP/1.1" 200 152 "-" "kube-probe/1.28" (0.030 118 0.030)
gunicorn-web stdout | 2024-04-22 04:26:30,661 [286] [INFO] [gunicorn.access] 10.129.2.2 - - [22/Apr/2024:04:26:30 +0000] "GET /health/instance HTTP/1.0" 200 152 "-" "kube-probe/1.28"
gcworker stdout | 2024-04-22 04:26:31,842 [67] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
gcworker stdout | 2024-04-22 04:26:31,843 [67] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:27:01.842666+00:00 (in 29.999545 seconds)
gcworker stdout | 2024-04-22 04:26:31,843 [67] [INFO] [apscheduler.executors.default] Running job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:27:01 UTC)" (scheduled at 2024-04-22 04:26:31.842666+00:00)
gcworker stdout | 2024-04-22 04:26:31,843 [67] [DEBUG] [peewee] ('SELECT DISTINCT "t1"."removed_tag_expiration_s" FROM "user" AS "t1" LIMIT %s', [100])
gcworker stdout | 2024-04-22 04:26:31,857 [67] [DEBUG] [peewee] ('SELECT "candidates"."repository_id" FROM (SELECT DISTINCT "t1"."repository_id" FROM "tag" AS "t1" INNER JOIN "repository" AS "t2" ON ("t1"."repository_id" = "t2"."id") INNER JOIN "user" AS "t3" ON ("t2"."namespace_user_id" = "t3"."id") WHERE ((((NOT ("t1"."lifetime_end_ms" IS %s) AND ("t1"."lifetime_end_ms" <= %s)) AND ("t3"."removed_tag_expiration_s" = %s)) AND ("t3"."enabled" = %s)) AND ("t2"."state" != %s)) LIMIT %s) AS "candidates" ORDER BY Random() LIMIT %s OFFSET %s', [None, 1712550391857, 1209600, True, 3, 500, 1, 0])
gcworker stdout | 2024-04-22 04:26:31,863 [67] [DEBUG] [__main__] No repository with garbage found
gcworker stdout | 2024-04-22 04:26:31,863 [67] [DEBUG] [data.database] Disconnecting from database.
gcworker stdout | 2024-04-22 04:26:31,863 [67] [INFO] [apscheduler.executors.default] Job "GarbageCollectionWorker._garbage_collection_repos (trigger: interval[0:00:30], next run at: 2024-04-22 04:27:01 UTC)" executed successfully
securityscanningnotificationworker stdout | 2024-04-22 04:26:31,933 [92] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
securityscanningnotificationworker stdout | 2024-04-22 04:26:31,934 [92] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:27:29.931002+00:00 (in 57.996834 seconds)
securityscanningnotificationworker stdout | 2024-04-22 04:26:31,934 [92] [INFO] [apscheduler.executors.default] Running job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:31 UTC)" (scheduled at 2024-04-22 04:26:31.933721+00:00)
securityscanningnotificationworker stdout | 2024-04-22 04:26:31,934 [92] [DEBUG] [workers.queueworker] Running watchdog.
securityscanningnotificationworker stdout | 2024-04-22 04:26:31,934 [92] [INFO] [apscheduler.executors.default] Job "QueueWorker.run_watchdog (trigger: interval[0:01:00], next run at: 2024-04-22 04:27:31 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:26:32,431 [77] [DEBUG] [apscheduler.scheduler] Looking for jobs to run
notificationworker stdout | 2024-04-22 04:26:32,431 [77] [DEBUG] [apscheduler.scheduler] Next wakeup is due at 2024-04-22 04:26:42.430973+00:00 (in 9.999530 seconds)
notificationworker stdout | 2024-04-22 04:26:32,431 [77] [INFO] [apscheduler.executors.default] Running job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:42 UTC)" (scheduled at 2024-04-22 04:26:32.430973+00:00)
notificationworker stdout | 2024-04-22 04:26:32,431 [77] [DEBUG] [workers.queueworker] Getting work item from queue.
notificationworker stdout | 2024-04-22 04:26:32,432 [77] [DEBUG] [peewee] ('SELECT "t1"."id", "t1"."queue_name", "t1"."body", "t1"."available_after", "t1"."available", "t1"."processing_expires", "t1"."retries_remaining", "t1"."state_id" FROM "queueitem" AS "t1" INNER JOIN (SELECT "t1"."id" FROM "queueitem" AS "t1" WHERE (((("t1"."available_after" <= %s) AND (("t1"."available" = %s) OR ("t1"."processing_expires" <= %s))) AND ("t1"."retries_remaining" > %s)) AND ("t1"."queue_name" ILIKE %s)) LIMIT %s) AS "j1" ON ("t1"."id" = "j1"."id") ORDER BY Random() LIMIT %s OFFSET %s', [datetime.datetime(2024, 4, 22, 4, 26, 32, 431674), True, datetime.datetime(2024, 4, 22, 4, 26, 32, 431674), 0, 'notification/%', 50, 1, 0])
notificationworker stdout | 2024-04-22 04:26:32,457 [77] [DEBUG] [workers.queueworker] No more work.
notificationworker stdout | 2024-04-22 04:26:32,457 [77] [DEBUG] [data.database] Disconnecting from database.
notificationworker stdout | 2024-04-22 04:26:32,457 [77] [INFO] [apscheduler.executors.default] Job "QueueWorker.poll_queue (trigger: interval[0:00:10], next run at: 2024-04-22 04:26:42 UTC)" executed successfully
notificationworker stdout | 2024-04-22 04:26:36,377 [77] [DEBUG] [util.metrics.prometheus] pushed registry to pushgateway at http://localhost:9091 with grouping key {'host': 'quayregistry-quay-app-57fc58dcd5-7cbxs', 'process_name': 'notificationworker.py', 'pid': '77'}